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I . Executive Summary 



A. Synopsis of the Disturbance and System Recovery 

On the afternoon of September 8, 2011 an 11-minute system disturbance 
occurred in the Pacific Southwest leading to cascading outages and leaving 
approxi mately 2.7 mi 1 1 ion customers without power. 1 The outages affected parts of 
Arizona, Southern California, and Baja California, Mexico. All of the San Diego area lost 
power, with nearly one-and-a-half million customers losing power, some for up to 12 
hours. The disturbance occurred near rush hour, on a business day, snarling traffic for 
hours. Schools and businesses closed, some flights and public transportation were 
disrupted, water and sewage pumpi ng stations lost power, and beaches were closed due 
to sewage spi I Is. M i 1 1 ions went without ai r conditioni ng on a hot day. 

The loss of a single 500 ki I ovol t (kV) 2 transmission lineinitiated the event, but 
was not the sole cause of the widespread outages. The system is designed, and should be 
operated, to withstand the loss of a single line, even one as large as 500 kV. The affected 
line-^Arizona Public Service's (APS) Hassayampa-N. Gila 500 kV line(H-NG)— is a 
segment of the Southwest Power Link (SWPL), a major transmission corridor that 
transports power in an east-west direction, from generators in Arizona, through the 
service territory of I mperial I rrigation District (I I D), into the San Diego area. It had 
tripped on multiple occasions, as recently asj uly 7, 2011 without causing cascading 
outages. 

With the SWPL's major east-west corridor broken by the loss of H-NG, power 
flows i nstantaneously redistri buted throughout the system, i ncreasi ng flows through 
lower voltage systems to the north of the SWPL, as power conti nued to flow i nto San 
Diego on a hot day during hours of peak demand. Combined with lower than peak 



"Customers" are not the same as "people" in utility parlance. The term customer generally refers to a single 
meter, whether at a residence, an apartment building, or a factory. Thus, a single customer could represent one 
or more persons, and a single person could be two customers, for example, if the same utility served both an 
individual's residence and his small business. Estimates of "people" affected by blackouts generally are prepared 
by increasing the customer numbers by a multiplier, often two or three. 

A list of acronyms used in this report is included in Appendix A. 
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generation levels in San Diego and Mexico, 3 this instantaneous redistribution of power 
flows created sizeable voltage deviations and equipment overloads to the north of the 
SWPL. Significant overloading occurred on three of 1 1 D's 230/92 kV transformers 
located at the Coachel la Val ley (CV) and Ramon substations, as wel I as on Western 
Electricity Coordinating Council (WECC) Path 44, 4 located south of the San Onofre 
Nuclear Generating Station (SONGS) in Southern California. 

The flow redistributions, voltage deviations, and resulting overloads had a ripple 
effect, as transformers, transmission lines, and generating units tripped offline, initiating 
automatic load shedding throughout the region in a relatively short time span. J ust 
seconds before the blackout, Path 44 carried all flows into the San Diego area as well as 
parts of Arizona and Mexico. Eventually, the excessive loading on Path 44 initiated an 
i nterti e separati on scheme at SON GS, desi gned to separate SDG&E from SCE . The 
SONGS separation scheme separated SDG&E from Path 44, led to the loss of the SONGS 
nuclear units, and eventually resulted in the complete blackout of San Diego and 
Comision Federal deElectricidad'sfCFE) Baja California Control Area. During the 11 
minutes of the event, the WECC Reliability Coordinator (WECC RC) issued no directives 
and only limited mitigating actions were taken by the Transmission Operators (TOPs) of 
the affected areas. 

Asa result of the cascading outages stemming from thisevent, customers in the 
SDG&E, 1 1 D, APS, Western Area Power Ad mini strati on- Lower Colorado (WALC), and 
CFE territories lost power, some for multiple hours extending into the next day. 
Specifically, 

• SDG&E lost 4,293 M egawatts (M W) of firm load, affecting approximately 14 million customers. 

• CFE lost 2,150 MW of net firm load, affecting approximately 11 mi 1 1 ion customers. 5 

• 1 1 D lost 929 M W of fi rm load, affecti ng approxi mately 146,000 customers. 



* Total summer peak generation for San Diego Gas and Electric's (SDG&E) territory and Comision Federal de 
Electricidad's (CFE) Baja California Control Area is 5,774 MW. On September 8, 2011, the total generation for 
SDG&E and CFE's Baja California Control Area was 4,168, a difference of 1,606 MW. 

4 Path 44 is one of 81 Rated Paths in the WECC region. A Rated Path is composed of "an individual transmission 
line or a combination of parallel transmission lines." WECC 2011 Path Rating Catalog, January 2011, at item 1-i. Path 
44, also referred to as "South of SONGS," is an aggregation of five 230 kV lines that delivers power in a north- 
south direction from the Southern California Edison (SCE) footprint in the Los Angeles area into the SDG&E 
footprint. 

5 CFE is Mexico's state-owned utility. Only its Baja California Control Area was affected on September 8, 2011. The 
inquiry is particularly grateful to CFE for its willingness to share data and information to assist the inquiry in 
developing the most accurate conclusions and recommendations. 
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APS lost 389 M W of firm load, affecting approximately 70,000 customers. 

WALC lost 74 M W of fi rm load, 64 M W of which affected APS's customers. The remai ni ng 10 M W 
affected 5 WALC customers. 



After the blackout the affected entities promptly instituted their respective 
restoration processes. 6 Al I of the affected entities had access to power from thei r own or 
neighboring systems and, therefore, did not need to use "black start" plans. 7 Although 
there were some delays in the restoration process due to communication and 
coordination issues between entities, the process was generally effective. SDG&E took 12 
hours to restore 100%of its load, and CFE took 10 hours to restore 100% of its load. 1 1 D, 
APS, and WALC restored power to 100% of their customers in approximately 6 hours. 
The affected entities also worked to restore generators and transmission lines that 
tripped during the event. 1 1 D and APS restored generation— 333 MW for 1 1 D and 76 MW 
for APS— in 5 hours. Meanwhile, CFE restored X915 MW of tripped generation in 56 
hours; SDG&E restored 2,229 MW of tripped generation in 39 hours; and SCE restored 
2,428 M W of tripped generation in 87 hours. 1 1 D restored its 230 kV transmission 
system in 12 hours and its 161 kV system in 9 hours; APS restored H-NG in 2 hours; 
SDG&E restored its 230 kV system in 12 hours; WALC restored its 161 kV system in 15 
hours; and CFE restored its 230 kV system in 13 hours and its 115 kV system in 10 hours. 

B. Map of Affected Area and Key Facilities Involved in the Event 

The fol lowi ng map, showi ng the areas affected by the September 8th event and 
the key facilities involved during the event, can be used as a reference throughout the 
report: 



" The term "affected entities" in this report refers to TOPs and Balancing Authorities (BAs) that were affected by 
the event. The affected entities include SDG&E, IID, APS, WALC, SCE, CFE, and the California Independent System 
Operator (CAISO). 

^ Black start plans work to energize systems using internal generation to get from shutdown to operating 
condition without assistance from the Bulk Electric System (BES). 
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C. Key Findings, Causes, and Recommendations 5 

The September 8, 20 IX event showed that the system was not bei ng operated i n a 
secure N-lstate. 9 This failure stemmed primarily from weaknesses in two broad areas- 
operations planning and real-time situational awareness— which, if done properly would 
have allowed system operators to proactively operate the system in a secure N-lstate 
during normal system conditions and to restore the system to a secure N-lstate as soon 
as possible, but no longer than 30 minutes. Without adequate planning and situational 
awareness, entities responsible for operating and overseeing the transmission system 
could not ensure reliable operations within System Operating Limits (SOLs) or prevent 
cascading outages in the event of a single contingency. 30 As demonstrated in Appendix 
C, inadequate situational awareness and planning were also identified as causes of the 
2003 blackout that affected an estimated 50 million people in the United States and 
Canada. 

Theinquiry also identified other underlying factors that contributed to the event, 
including: (1) not identifying and studying the impact on Bulk-Power System (BPS) u 



° While this section highlights the most significant causes, findings, and recommendations, the report details the 
complete list of findings, causes, and recommendations in section IV. In addition, for ease of reference all of the 
findings and recommendations are summarized in table format in Appendix B. 

^The North American Electric Reliability Corporation's (NERC) mandatory Reliability Standards applicable to the 
BES require that the BES be operated so that it generally remains in a reliable condition, without instability, 
uncontrolled separation or cascading, even with the occurrence of any single contingency, such as the loss of a 
generator, transformer, or transmission line. This is commonly known as the "N-1 criterion." N-1 contingency 
planning allows entities to identify potential N-1 contingencies before they occur and to adopt mitigating 
measures, as necessary, to prevent instability, uncontrolled separation, or cascading. As the Federal Energy 
Regulatory Commission (Commission) stated in Order No. 693 with regard to contingency planning, "a single 
contingency consists of a failure of a single element that faithfully duplicates what will happen in the actual 
system. Such an approach is necessary to ensure that planning will produce results that will enhance the 
reliability of that system. Thus, if the system is designed such that failure of a single element removes from 
service multiple elements in order to isolate the faulted element, then that is what should be simulated to assess 
system performance." Mandatory Reliability Standards for the Bulk Power System, Order No. 693, FERC Stats. & 
Regs. H 31,242, at P 1716 (2007), order on reh'g, Mandatory Reliability Standards for the Bulk-Power System, 120 
FERC 11 61,053 (Order No. 693-A) (2007). 

^ A contingency is the unexpected failure of an electrical system component. 

^■The BPS is defined by Section 215(a) (1) of the Federal Power Act as "facilities and control systems necessary 
for operating an interconnected electric energy transmission network (or any portion thereof), and electric 
energy from generating facilities needed to maintain transmission system reliability." The meaning of BPS and 
BES differ somewhat and, thus, this report uses each term in its proper context. With respect to reliability, the 
Commission has jurisdiction over all users, owners, and operators of the BPS. In Order No. 693 at P 75, the 
Commission adopted, at least for an initial period, the BES definition as the threshold for application of the NERC 
Reliability Standards. Thus, this report uses BES when referring to entities' specific facilities or elements that are 
subject to the Reliability Standards, but BPS when discussing the overall reliability impact. On January 25, 2012, 
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reliability of sub- 100 kV facilities in planning and operations; 12 (2) thefailureto 
recognize I nterconnection Reliability Operating Limits (I ROLs) in the Western 
I nterconnection; 13 (3) not studying and coordinating the effect of protection systems, 
including Remedial Action Schemes (RASs), during plausible contingency scenarios; 14 
and (4) not providing effective tools and operating instructions for use when reclosing 
lines with large phase angle differences across the reclosing breakers. 15 

With regard to operations planning, some of the affected entities' seasonal, next- 
day, and real-time studies do not adequately consider: (1) operations of facilities in 
external networks, including the status of transmission facilities, expected generation 
output, and load forecasts; (2) external contingencies that could impact their systems or 
internal contingencies that could impact their neighbors' systems; and (3) the impact on 
BPS reliability of internal and external sub- 100 kV facilities. Asa result, these entities' 
operations studies did not accurately predict the impact of the loss of APS's H-NG or the 
loss of 1 1 D's three 230/ 92 kV transformers. I f the affected entities had more accurately 
predicted the impact of these losses prior to the event, these entities could have taken 
appropriate pre-contingency measures, such as dispatching additional generation to 
mitigate overloads and prevent cascading outages. 

To improve operations planning in the WECC region, this report makes several 
recommendations designed to ensure that TOPs and BAs, 16 as appropriate: (1) obtain 
information on the operations of neighboring BAs and TOPs, including transmission 
outages, generation outages and schedules, load forecasts, and scheduled interchanges; 

NERC filed a petition with the Commission for approval of a revised definition of the BES. The proposed definition 
of BES would cover all elements operated at 100 kV or higher, with a list of specific inclusions and exclusions. 
Transformers with the primary terminal and at least one secondary terminal operated at 100 kV or higher are on 
the list of specific inclusions. See North American Electric Reliability Corp., Docket No. RM12-6-000. This report 
takes no position on the petition. 

12 

M This report does not attempt to define the limits of which sub-100 kV facilities impact BPS reliability. Certainly, 
many facilities below 100 kV do not impact BPS reliability. The sub-100 kV facilities in this event affected the BPS 
because they were in parallel to significant transmission corridors. 

13 

" This report recommends that WECC RC should work with TOPs to consider whether any SOLs in the Western 
Interconnection constitute I ROLs. As part of this effort, WECC RC should: (1) work with affected TOPs to 
consider whether Path 44 and H-NG should be recognized as IROLs; and (2) validate existing SOLs and ensure that 
they take into account all transmission and generation facilities and protection systems that impact BPS reliability. 

^ This failure caused the derived SOLs on H-NG and Path 44 to be invalid on the day of the event. 

^ As discussed in more detail in connection with Finding and Recommendation 27 below, when a line trips, the 
phase angle at one end of the line may be much larger than the phase angle at the other end. If the difference 
between the two angles is too great, reclosing the line could cause damage to generators or even system 
instability. 

Ifi 

See "Reliability Responsibilities" section at page 16 below. 
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(2) identify and plan for external contingencies that could impact their systems and 
internal contingencies that could impact their neighbors' systems; and (3) consider 
facilities operated at less than 100 kV that could impact BPS reliability. Thiseffort 
should include a coordinated review of planning studies to ensure that operation of the 
affected Rated Paths will not result in the loss of non-consequential load, system 
instability, or cascading outages, with voltage and thermal limits within applicable 
ratings for N-l contingencies originating from within or outside an entity's footprint. 

The September 8th event also exposed entities' lack of adequate real-time 
situational awareness of conditions and contingencies throughout the Western 
I nterconnection. For example, many entities' real -time tools, such as State Estimator 
and Real-Time Contingency Analysis (RTCA), are restricted by models that do not 
accurately or ful ly reflect faci I ities and operations of external systems to ensure 
operation of the BPS in a secure N-l state. Also, some entities' real-timetoolsarenot 
adequate or operational to alert operators to significant conditions or potential 
contingencies on their systems or neighboring systems. The lack of adequate situational 
awareness limits entities' ability to identify and plan for the next most critical 
conti ngency to prevent i nstabi I ity, uncontrol led separation, or cascadi ng outages. I f 
some of the affected entities had been aware of real-time external conditions and run (or 
reviewed) studies on the conditions prior to the onset of the event, they would have been 
better prepared for the i mparts when the event started and may have avoided the 
cascadi ng that occurred. 

To improve situational awareness in theWECC region, this report makes several 
recommendations: (1) expand entities' external visibility in their models through, for 
example, more complete data sharing; (2) improve the use of real -time tools to ensure 
the constant monitoring of potential internal or external contingencies that could affect 
reliable operations; and (3) improve communications among entities to help maintain 
situational awareness. I n addition, TOPs should review their real-time monitoring tools, 
such as State Estimator and RTCA, to ensure that such tools represent critical facilities 
needed for the reliable operation of the BPS. These improvements will enable system 
operators to uti I ize real-ti me operati ng tools to proarti vely operate the system i n a secure 
N-l state. 

I n addition to the planning and situational awareness issues, several other factors 
contributed to the September 8th event. For example, WECC RC and affected entities do 
not consistently recognize the adverseimpart that sub- 100 kV faci I ities can have on BPS 
reliability. The prevailing SOLs should have included the effects of faci I ities that had not 
been identified and classified as part of the BE S, as well as the effects of critical facilities 
such as Special Protection Systems (SPSs) and the SONGS separation scheme. Relevant 
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to the event these entities did not consider 1 1 D's 92 kV network and facilities, including 
the CV and Ramon 230/ 92 kV transformers, as part of the BES, despite some previous 
studi es i ndi cati ng thei r i mpact on the BPS due to the fact they were el ectri cal ly i n 
parallel with hi gher-voltage facilities. 17 If these facilities had been designated as part of 
the BES, or otherwise incorporated into planning and operations studies and actively 
monitored and alarmed in RTCA systems, the cascading outages may have been avoided. 
Accordingly, theinquiry makes a recommendation to ensure that facilities that can 
impact BPS reliability, regardless of voltage level, are considered for classification as part 
of the BES and/ or studied as part of entities' planning in various time horizons. 

The inquiry also found some significant issues with protection system settings 
and coordination. For example, 1 1 D used conservative overload relay trip settings on its 
CV transformers. The relays were set to tri p at 127% of the transformers' normal rati ng, 
which is just above the transformers' emergency rating (110% of normal rating). Such a 
narrow margin between the emergency rating and overload trip setting resulted in the 
faci I i ties bei ng automatical ly removed from service without providi ng operators enough 
time to mitigate the overloads. As a result of these settings, both CV transformers 
tripped within 40 seconds of H-NG tripping, initiating cascading outages. To avoid a 
similar problem in the future, the inquiry recommends that 1 1 D and other Transmission 
Owners (TOs) review their transformers' overload protection relay settings. A good 
guidelinefor protective relay settings is Reliability Standard PRC-023-1R111 which 
states that rel ays be "set to al I ow the transformer to be operated at an over I oad I evel of at 
least 150% of the maxi mum appl i cable nameplate rati ng, or 115% of the highest operator 
established emergency transformer rating, whichever is greater." TOPs should also plan 
to take proper pre-contingency mitigation measures with due consideration for the 
applicable emergency ratings and overload protection settings (M W and time del ay) 
before a faci I ity loads to its relay tri p poi nt and is automatical ly removed from service. 

The SONGS separation scheme's operation provides another example of the lack 
of studies on, and coordination of, protection systems. This scheme, classified by SCE as 
a "Safety Net," 18 had a significant impact on BPS reliability, separating SDG&E from 



A/ See, e.g., CFE's Path 45 Increase Rating Phase 2 Study Report, January 12, 2011, at 19. 
10 

A Safety Net protection system protects the power system from unexpected, low-probability events that are 
outside the normal planning criteria, but which may lead to a complete system collapse. Safety Nets operate to 
minimize the severity of the event and attempt to prevent a system collapse or cascading outages. A Safety Net 
is typically intended to handle severe disturbances resulting from extreme, though perhaps not well-defined, 
events. A Safety Net is subject to review by the WECC Remedial Action Scheme Reliability Subcommittee if 
unintended operation would result in cascading or other performance standard violations. WECC Guideline: 
Remedial Action Scheme Classification, February 9, 2009. 
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SCE, resulting in the loss of both SONGS nuclear generators, and blacking out SDG&E 
and CFE. Nevertheless, none of the affected entities, including SCE, as the owner and 
operator of the scheme, studied its impact on BPS reliability. The September 8th event 
shows that all protection systems and separation schemes, including Safety Nets, RASs, 
and SPSs, should be studied and coordinated periodically to understand their impact on 
BPS reliability to ensure their operation, inadvertent operation, or misoperation does not 
have unintended or undesirable effects. 
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II. Introduction 



A. Inquiry Process 

On September 9, 201X the Commission and NERC jointly announced an inquiry 
to determi ne the causes of the outages and make recommendations for preventi ng such 
events i n the future. The purpose of the i nqui ry was not to determi ne whether there may 
have been violations of applicable regulations, requirements, or standards subject to the 
Commission's jurisdiction. Thus, whilethis report describes conduct which may warrant 
future investigations under Part lb of the Commission's regulations, 19 or actions by 
NERC under its Compliance Monitoring and Enforcement Program, 20 it draws no 
conclusions about whether violations occurred. 

The inquiry was composed of smaller teams with particular subject-matter 
expertise, primarily from Commission and NERC professional staff, each of which 
conducted rigorous analyses of a key issue or issues involved in the event. Those teams 
and thei r pri mary responsi bi I i ti es were as fol I ows: 

• Sequence of Events - developed a precise and accurate sequence of events (SOE) to provide a 
foundation for root cause analysis, computer model simulations, and other analytical aspects of the 
inquiry. 

• System Modeling and Simulation - developed an accurate system modeling case, 
benchmarked the case to actual conditions at critical times, replicated system conditions leading up 
to and duri ng the outage, and si mulated alternate "what if" scenarios. 

• Root Causeand Human Performance Analysis- performed in a systematic evaluation 
of the root causes and contri buti ng factors and identified areas requi ri ng further i nqui ry. 

• Operations Tools, Supervisory Control and Data Acquisition 

( SCAD A)/ Energy Management System (EMS), Communications, and 
Operations Planning- considered all aspects of the blackout related to operator and 
reliability coordinator knowledge of system conditions, actions or inactions, and communications, 
particularly the observability of the electric system and effectiveness of operational reliability 
assessment tools. 

• Frequency/ Area Control Error (ACE) Analysis- reviewed potential frequency 
anomalies related to the blackout, and analyzed underfrequency generator, load, and tie line 
tripping. 



^iSCF.R. Part 1b (2011). 
20 

v NERC Compliance Monitoring and Enforcement Program, Appendix 4C to the NERC Rules of Procedure, 
January 31, 2012. 
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• System Planning, Design, and Studies- analyzed factors used in setting SOLs and actual 
I i mits i n effect on the day of the blackout, determi ned whether those I i mits were exceeded, and 
analyzed the extent to which actual system conditions varied from the assumptions used in setting 
the SOLs. 

• Transmission and Generation Performance, Protection, Control, 
Maintenance, and Damage- analyzed the causes of automatic facility operations and 
generator trips, analyzed transmission and generation facility maintenance practices, and identified 
equipment damage. 

• Restoration Review - reviewed the appropriateness and effectiveness of the restoration plans 
i mplemented, as wel I as the effectiveness of the coordi nation of these plans among the affected 
entities and WECC RC. 

Each team not only examined its own subject area to determine what may have 
contributed to the event but also considered lessons learned and potential 
recommendations for preventing such events in the future. 

The inquiry devoted substantial time and resources to determine and study the 
causes of the event and develop meaningful recommendations with the goal of 
preventi ng si mi I ar events i n the future. The team's analyses were extensive, i nvol vi ng 
the review of high-quality data from various reliability entities in the WECC region and 
simulations of the event using sophisticated computer models. Described below in 
summary form are the primary steps the inquiry took to complete its analysis. 

Data Gathering 

The inquiry received and reviewed more than 20 gigabytes of data from 
approximately 500 data requests sent to entities in and around the affected areas. On 
September 19, 2011 the inquiry also began site visits with various entities involved in the 
outages, including entities with responsibility for balancing load and generation, 
transmission operation, and reliability coordi nation. During the site visits, the inquiry 
toured control centers, conducted dozens of interviews and depositions, and viewed 
equi pment i nvol ved i n the event. These vi sits and depositi ons al I owed the i nqui ry to 
learn about control room operations and practices, system status and conditions on the 
day of the event, operating procedures, planning, operations, and real -time tools, and 
restoration planning and procedures. The inquiry also conducted dozens of follow-up 
meetings and issued follow-up data requests. 

Of particular use to the inquiry were phasor measurement unit (PMU) records. 
PM Us are complex, multi-functional, high resolution recording devices installed widely 
throughout the Western I nterconnection pursuant to a voluntary WECC-wide initiative. 
PMUs provide continuous, high-speed (30 scans per second) records of system 
conditions, including frequency, voltage, and phase angle relations. Thecontinuous 
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nature of the data avai I abl e through the P M U s, as wel I as thei r wi de d i str i buti on 
throughout the power system, proved especial ly val uable to the i nqui ry i n formi ng an 
accurate picture of the SOE and state of the system at particular poi nts i n ti me 
throughout the disturbance. 

SOE Methodology 

More than 100 notable events occurred in less than 11 minutes on Septembers, 
2011 The i nqui ry's SOE team establ ished a precise and accurate sequence of outage- 
related events to form a critical bui Idi ng block for the other parts of the i nqui ry. 1 1 
provided, for example, a foundation for the root cause analysis, computer-based 
simulations, and other event analyses. Although entities time-stamp much of the data 
related to specific events, their time- stamping methodologies vary, and not all of the 
time-stamps were synchronized to the National I nstituteof Standards and Technology 
(NIST) standard clock in Boulder, Colorado. Validating the precise timing of specific 
events became a time-consuming, important, and sometimes difficult task. The 
availability of global positioning system (GPS)-ti me synchronized PMU data on 
frequency, voltage, and related power angles made this task much easier than in previous 
blackout inquiries and investigations. 

To develop the SOE, the SOE team started by resolving discrepancies between the 
multiple sources of data, sign convention inconsistencies, and incorrect data. The SOE 
team then developed an events database starting with all known events and times. 
Initial sources for the development of the database included preliminary reports filed by 
the affected entities as well asinitial responses to data requests. Theteamthen 
exami ned each record i n the database to verify event ti mes usi ng avai I able SCAD A and 
PM U data. As the frequency, lineflow, or voltage data suggested that additional events 
might have occurred on the system, the team added other possible events and verified 
them through additional data requests. 

The SOE team developed multiple iterations of an SOE narrative document based 
on the database and the avai lable SCADA and PM U data. Some iterations of the SOE 
narrative required that more data be requested of affected entities, and ultimately 
multi pie data requests were sent to each entity. After the team completed the SOE 
narrative, the inquiry's Modeling and Simulation team verified the SOE using power 
flow, voltage stability, and dynamic stability analyses. 
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Power Flow and Dynamics Analysis 

The i nqu i ry's M odel i ng and Si mul ati on team, after val i dati ng the SOE , 
considered several "what if" scenarios. The Modeling and Simulation team's work is 
described in more detail in Appendix D. Power flow analyses study power systems under 
quasi-steady-state conditions by matching load and generation to obtain voltage 
magnitude and angle at each bus and the real and reactive power flowing through each 
transmission facility. Dynamic stability analyses study the impact of disturbances on 
frequency, voltage, and rotor angle stability, and determine whether transients in the 
power system are stable, thus al lowi ng the power system to return to a quasi -steady- state 
operating condition following a disturbance. 21 

As the first step in performing power flow and dynamic stability analyses, the 
Modeling and Simulation team developed and benchmarked a modeling case of system 
conditions prior to the event. The team started with theWECC heavy summer base case 
and made adjustments based on State Estimator snapshots, EMS data, actual generation 
and schedules, PM U data, and a base case prepared by a separate team (led by CAI SO) 
that studied the event. The team further adjusted and benchmarked the base case using 
SCAD A and PMU data to match the system conditions for the entire event. The team 
devoted considerable time and effort to resolving discrepancies between the various 
sources of data to best cal i brate the model i ng case to actual measured data. As 
illustrated by Figure! on the next page, and described in more detail in Appendix D, 
the Modeling and Simulation team achieved a significant degree of accuracy. This figure 
compares Path 44 flows simulated by the Modeling and Simulation team to actual Path 
44 PMU data. 

After developing and benchmarking a valid case, the Modeling and Simulation 
team simulated the entire SOE using both power flow and dynamic simulations. This 
replication of the SOE established the validity of the model and enabled meaningful 
simulation of several alternative scenarios, developed to answer "what if" questions 
regarding the event. For example, the inquiry considered what would have happened if 
some of the affected entities had dispatched generation at certain locations during the 
event, if overload relays had been set at different levels, or if RASs, Safety Nets, or other 
SPSs had been designed or operated differently. 



Transient stability refers to the ability of synchronous generators to move to a new quasi-steady-state 
operating point while remaining synchronized after the system experiences a disturbance. 
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Figure 1: Comparison of Actual and Simulated Path 44 Flows 
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Outreach Sessions 



After devel opi ng a I i st of prel i mi nary f i nd i ngs and recommendati ons, the i nqu i ry 
conducted outreach meetings with various industry associations and groups, including 
CAI SO, WECC, the American Public Power Association (APPA), the North American 
Transmission Forum (NATF), the Edison Electric I nstitute(EEI), the National Rural 
Electric Cooperative Association (NRECA), and representatives from Regional Entities 
(REs), Regional Transmission Organizations, and I ndependent System Operators. Team 
members shared the inquiry's preliminary findings and recommendations on a non- 
public basis with members of these organizations to obtain feedback and, with respect to 
the recommendations, input as to their practicality and feasibility. The inquiry 
considered thefeedback and input provided by these organizations and incorporated 
much of it into the findings and recommendations included in this report. 
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B. System Overview 

This subsection provides an overview of: (1) the Western I nterconnection and its 
position in the North American electric grid; (2) the reliability entities responsible for 
operating the grid; (3) a description of the affected entities; and (4) a discussion of the 
interconnected nature of these entities. 

The Western Interconnection and Its Position in the North American 
Electric Grid 



NERC shares its mission of ensuring the reliability of the BPS in North America 
with eight REs through a series of delegation of authority agreements. WECC is the 
designated RE responsible for coordinating and promoting BPS reliability in the Western 
Interconnection. In its capacity as the RE, WECC monitors and enforces compliance 
with Reliability Standards by the users, owners, and operators of the BPS. WECC also 
functions as an I nterconnection- wide planning facilitator, aiding in transmission and 
resource integration planning at the request of its members, as well as a provider of data, 
analysis, and studies related to transmission planning and reliability issues. 

The WE CC regi on extends from Canada to M exi co. 1 1 i ncl udes the provi nces of 
Alberta and British Columbia, the northern portion of Baj a California, Mexico, the states 
of Washington, Oregon, California, Idaho, Nevada, Utah, Arizona, Colorado, Wyoming, 
and portions of M ontana, South Dakota, New M exico, and Texas. See Figure 2, on the 
next page. The WECC region is nearly 18 million square miles in size, has over 126,000 
miles of transmission, and serves a population of 78 million. WECC contains 37 BAsand 
53 TOPs. Due to the diverse characteristics of this extensive region, WECC encounters 
unique challenges in day-to-day coordination of its interconnected system. WECC istied 
to the Eastern Interconnection through a number of hi gh- voltage direct current 
transmission ties. 

WECC also operates two RC offices that provide situational awareness and real- 
time monitoring of the entire Western Interconnection. WECC RC was an affected 
entity, and will be discussed with other affected entities below. 
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Figure 2: Map of WECC Region 




Reliability Responsibilities 

NERC categorizes the entities responsible for planning and operating the BPS in 
a reliable manner into multiplefunctional entity types. The NERC functional entity 
types most relevant to this event are BAs, TOs, TOPs, Generator Operators (GOPs), 
Planning Coordinators (PCs), Transmission Planners (TPs), and RCs. These functions 
are described in more detail in NE RCs Reliability Functional Model. 22 Some of the 
affected entiti es conduct multi pie rel i abi I ity fundi ons. 

■ Balancing Authority 

The BA integrates resource plans ahead of time, maintains in real time the 
balance of electricity resources (generation and interchange) and electricity demand or 
load within its footprint, and supports the I nterconnection frequency in real time. There 



" NERC Reliability Functional Model, Version 5, 
http://www.nerc.com/files/Functional_JVlodel_Vs_Final_2009Dec1.pdf. 
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are 37 BAs i n the WECC footpri nt. The fol lowi ng five BAs were affected by the event: 
APS, 1 1 D, WALC, CAI SO, and CFE. 

■ Transmission Owner, Transmission Operator and Generator Operator 

The TO owns and maintains transmission facilities. The TOP is responsible for 
the real -time operation of the transmission assets under its purview. TheTOP has the 
authority to take corrective actions to ensure that its area operates reliably. TheTOP 
performs reliability analyses, including seasonal and next-day planning and RTCA, and 
coordinates its analyses and operations with neighboring BAs and TOPs to achieve 
reliable operations. It also develops contingency plans, operates within established 
SOLs, and monitors operations of the transmission facilities within its area. There are 53 
TOPs in the WECC region. The fol I owing seven TOPs were affected by the event: APS, 
1 1 D, WALC, CAI SO, CFE, SDG&E, and SCE. The GOP operates generating unit(s) and 
performs the functions of supplying energy and other services required to support 
reliable system operations, such as providing regulation and reserve capacity. 

■ Planning Coordinator 

The PC i s responsi bl e for coord i nati ng and i ntegrati ng transmi ssi on f aci lity and 
service plans, resource plans, and protection systems. 23 

■ Transmission Planner 

TheTP is responsible for developing a long-term (generally one year and beyond) 
plan for the reliability of the interconnected bulk transmission systems within its portion 
of the Planning Coordinator Area. 

■ Reliability Coordinator 

The RC and TOP have similar roles, but different scopes. TheTOP directly 
maintains reliability for its own defined area. The RC is the "highest level of authority" 
according to NERC, and maintains reliability for the Interconnection as a whole. Thus, 
the RC is expected to have a "wide-area" view of the enti re I nterconnection, beyond what 
any singleTOP could observe, to ensure operation within I ROLs. 

The RC oversees both transmission and balancing operations, and it has the 
authority to direct other functional entities to take certain actions to ensure reliable 



PCs are the same as Planning Authorities (PAs) with respect to NERC registration and the Reliability Standards. 
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operation. The RC, for example, may di red a TOP to take whatever action is necessary 
to ensure that I ROLs are not exceeded. 24 The RC performs reliability analyses including 
next-day planning and RTCAfor the Interconnection, but these studies are not intended 
to substitute for TOPs' studies of thei r own areas. Other responsi bi I ities of the RC 
i ncl ude respondi ng to requests from TOPs to assi st i n mi ti gati ng equi pment overl oads. 
The RC also coordinates with TOPs on system restoration plans, contingency plans, and 
rel i abi I i ty- rel ated servi ces. 

Descriptions of Affected Entities 

The fol I owi ng enti ti es were affected by the September 8th event: 

■ WECC RC 

In its capacity as the RC, WECC isthehighest level of authority responsible for 
the reliable operation of the BPS in the Western I nterconnection. WECC RC oversees 
the operation of the Western Interconnection in real time, receiving data from entities 
throughout the entire I nterconnection, and providing high-level situational awareness 
for the enti re system. WECC RCcan direct the entities it oversees to take certain actions 
in order to preserve system reliability. Although WECC is both an RE and an RC, these 
two functions are organizationally separated. 

■ Imperial Irrigation District 

1 1 D, which encompasses the I mperial Val ley, the eastern part of Coachel la Val ley 
in Riverside County, and a small portion of San Diego County, in California, owns and 
operates generation, transmission, and distribution facilities in its service area to provide 
comprehensive electric service to its customers. Thus, 1 1 D is a vertically integrated 
utility. I ID's generation consists of hydroelectric units on the Ail-American Canal as well 
as oil-, nuclear-, coal-, and gas-fired generation facilities, with a total net capability of 
514 M W. 1 1 D purchases power from other electric uti I ities to meet its peak demands i n 
summer, which can exceed 990 M W. 1 1 D's transmission system consists of 
approximately X400 miles of 500, 230, 16\ and 92 kV lines, as well as 26 transmission 
substations. Among other NERC registrations, I ID is a TOP, BA, and TP responsible for 
resource and transmission planning, load balancing, and frequency support for its 
footprint. 



For example, IRO-005-1 R.5 requires that "[e]ach [RC] shall identify the cause of any potential or actual SOL or 
IROL violations. The [RC] shall initiate the control action or emergency procedure to relieve the potential or 
actual IROL violation without delay, and no longer than 30 minutes. The [RC] shall be able to utilize all resources, 
including load shedding, to address an IROL violation." 



- 18- 



FERC/NERC Staff Report on the September 8, 2011 Blackout 



■ Arizona Public Service 

APS is a vertically integrated utility that serves a 50,000 square mile territory 
spanning 11 of Arizona's 15 counties. Among other NERC registrations, APS is the TOP 
and BA for its territory. APS engages in both marketing and grid operation functions, 
which are separated. APS owns and operates transmission facilities at the 500 
(including H-NG), 345, 230, 115, and 69 kV levels, and owns approximately 6,300 MW 
of installed generation capacity. APS's 2011 peak load was 7,087 MW. 

■ Western Area Power Administration - Lower Colorado 

WALC is one of thefour entities constituting the Western Area Power 
Administration, a federal power marketer within the United States Department of 
Energy. WALC operates in Arizona, Southern California, Colorado, Utah, New Mexico, 
and Nevada, and is registered with NERC as a BA, TOP, and PC for its footprint. As a net 
exporter of energy, WALC's territory has over 6,200 M W of generation, serving at most 
2,100 MW of peak load. A majority of WALC's generation is federal hydroelectric 
facilities, with the balance consisting of thermal generation owned and operated by 
independent power producers. WALC also operates an extensive transmission network 
within its footprint, and is interconnected with APS, SCE, and nine other balancing 
areas. 

■ San Onofre Nuclear Generating Station 

SONGS is a two-unit nuclear generation facility capable of producing 
approximately 2,200 M W of power, and is located north of San Diego. 25 SONGS 
produces approxi mately 19% of the power used by SCE customers and 25% of the power 
used by SDG&E customers. SONGS is jointly owned by SCE (78.21%), SDG&E (20%), 
and the City of Riverside (179%). SCE, as TO and GO, is responsible for ensuring the 
safe and reliable operation of SONGS within the grid. 

■ California Independent System Operator 

CAI SO runs the pri mary market for wholesale electric power and open-access 
transmission in California, and manages the high-voltage transmission lines that make 



SONGS is currently in the midst of an extended outage. According to a March 2012 press release by CAISO, if 
both SONGS units remain offline for the summer, "San Diego and portions of the Los Angeles Basin may face local 
reliability challenges." http://www.caiso.com/Documents/SummerGridOutlook Complicated- 
PossibleExtendedOutage-NuclearPowerPlant.pdf. 
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up approximately 80% of California's power grid. CAI SO operates its market through 
day-ahead and hour-ahead markets, as well as scheduling power in real time as 
necessary. Among other registrations, CAI SO is PC and BA for most of California, 
including the city of San Diego. It also acts as TOP for several entities within its 
footprint, including SDG&E and SCE. CAI SO likewise engages in modeling and planning 
functions in order to ensure long-term grid reliability, as well as identifying 
infrastructure upgrades necessary for grid function. 

■ San Diego Gas and Electric 

SDG&E is a utility that serves both electricity and natural gas to its customers in 
San Diego County and a portion of southern Orange County, and is the primary utility 
for the city of San Diego. SDG&E owns relatively little generation— approximately 600 
MW— although generation owned by others in its footprint brings the total generation 
capacity of the area above 3,350 M W. Peak load for the area can exceed 4,500 M W i n 
the summer. SDG&E also operates an extensive high- voltage transmission network at 
the 500, 230, and 138 kV levels. SDG&E, operating as a TOP within CAISO's BA 
footprint, has delegated part of its responsibilities as aTOP to CAI SO. 

■ Comision Federal de Electricidad - Baja California Control Area 

CFE is the only electric utility in Mexico, servicing up to 98% of the total 
population. CFE's Baja California Control Area is not connected to the rest of M exico's 
electric grid but is connected to the Western Interconnection. CFE's Baja California 
Control Area covers the northwest corner of Mexico, including the cities of Tijuana, 
Rosarito, Tecate, Ensenada, Mexicali, and San Luis Rio Colorado. CFE's Baja California 
Control Area operates transmission systems at the 230, 16 X 115, and 69 kV levels, and 
owns 2,039 M W of gross generati ng capacity and the rights to a 489 M W i ndependent 
power producer within the Baja California Control Area. CFE's Baja California Control 
Area had a net peak load of 2,184 M W for summer 2010. CFE's Baja California Control 
Area is connected at the 230 kV level with SDG&E through two transmission lines on 
WECC Path 45. CFE functions as the TO, TOP, and BAfor its Baja California Control 
Area under the oversight of WECC RC. For the remainder of this report, "CFE" refers 
only to its Baja California Control Area. 

■ Southern California Edison 

SCE is a large investor-owned utility which provides electricity in central, coastal, 
and southern California. SCE is a wholly-owned subsidiary of Edison International, 
which is also based in California. Among other NERC registrations, SCE operates as a 
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TOP within CAI SO's BA footprint and has delegated part of its responsibilities as a TOP 
toCAISO. SCE is also registered as TP, and is responsibl e for the reli ability assessments 
of the SONGS separation scheme. SCE owns 5,490 circuit miles of transmission lines, 
including 500, 230, and 161 kV lines. SCE also operates a subtransmission system of 
7,079 circuit miles at the 115, 66, 55, and 33 kV levels. Of the affected entities, SCE is 
interconnected with APS, 1 1 D, and SDG&E at various transmission voltage levels. SCE 
owns over 5,600 MW of generation, including a majority share in SONGS, and its peak 
load exceeds 22,000 MW. Along with SONGS staff, SCE is responsible for the safe and 
rel i abl e operati on of the n ucl ear f aci I i ty. 

Interconnected Operations 



The September 8th event exempl ifies the i nterconnected operations of three 
parallel transmission corridors through which power flows into the area where the 
blackout occurred. Typically, BAs, through dispatch, balance the flows on these 
corridors so that no one corridor experiences overloads in an N-lsituation, but this did 
not happen on September 8th. 

The first transmission corridor consists of the 500 kV H-NG, which is one of 
several transmission lines forming Path 49 ("East of River"). Along with two 500 kV 
lines, one from North Gil a to Imperial Valley and another from Imperial Valley to 
Miguel, they form theSWPL. The majority of theSWPL is geographically parallel to the 
U nited States-M exico border. The SWPL meets the SDG&E and 1 1 D systems at the 
Imperial Valley substation. This is shown as the "H-NG Corridor" on Figure 3, on the 
next page. 

The second corridor is Path 44, also known as "South of SONGS," operated by 
CAI SO. Thiscorridor includes the five 230 kV linesin the northernmost part of the 
SDG&E system that connect SDG&E with SCE at SONGS. 

The third transmission corridor, shown as the "S Corridor" on Figure 3, consists 
of lower voltage (230, 161and 92 kV) facilities operated by 1 1 D and WALC in parallel 
with those of SCE, SDG&E, and APS. The only major interconnection between II D and 
SDG&E is through the 230 kV "S" Line, which connects the SDG&E/ 1 ID jointly-owned 
I mperial Valley Substation (operated by SDG&E) to 1 1 D's El Centra Switching Station. 
The S Line interconnects the southern 1 1 D system with SDG&E and APS at I mperial 
Valley, which is also a terminus for the SWPL segment from Miguel and theSWPL 
segment from North Gila. WALC is connected to the SCE system and the rest of the 
Western I nterconnection by 161 kV ties at Blythe, to 1 1 D by the 161 kV tie between 
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WALC's Knob and 1 1 D's Pilot Knob substations, and to APS by a 69 kV tie via Gila at 
North Gila. 

The eastern end of the SWPL, which terminates at APS's H assayampa hub, is 
connected to SCE viaa500 kV linethat connects APS's Palo Verde and SCE's Devers 
substations. The northern 1 1 D system is connected to SCE's Devers substation via a 230 
kV transmission line that connects from Devers to II D'sCV substation. These 
connections, along with SDG&E's connection to SCE via Path 44's terminus at SONGS, 
make the SWPL, Path 44, and 1 1 D's and WALC's systems operate as electrically parallel 
transmission corridors. 26 The foil owing simplified diagram illustrates the 
interconnected nature of these three parallel corridors. Red lines represent 500 kV, blue 
lines represent 230 kV, and green lines represent 161 kV. 



Figure 3: Three Parallel Corridors 




M Power transfers from APS to SDG&E and CFE generally flow across the SWPL, but, due to parallel path flows, 
also known as loop flows, some of the power transfers flow through 1 1 D's and WALC's systems. Loop flow refers 
to power flow along any transmission paths that are in parallel with the most direct geographic or contract path. 
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III. Sequence Of Events 27 



The 11 minutes of the disturbance are divided into seven phases, as highlighted in 
Figure4, on the next page. This figure displays the progressive loading of thefive230 
kV tie lines from SCE north of San Diego that form Path 44. This section describes how 
the loss of various elements during an 11- minute period combined to exceed the 8,000 
amp setting on the SONGS separation scheme. After sustained loading on Path 44 above 
8,000 amps, the SONGS separation scheme operated. Once the SONGS separation 
scheme operated, San Diego and 1 1 D, CFE, and Yuma, Arizona, blacked out in less than 
30 seconds. This section is divided into subsections for each phase, including the key 
events duri ng the phase, thei r causes and effects, and, where relevant, what the affected 
entities knew and did not know as the events were unfolding. Each section begins with a 
brief summary. A final subsection describes restoration efforts after the blackout. 

A set of graph i cs i s i ncl uded at the end of each phase to demonstrate the effect of 
the events during the phase. The first graphic in each set depicts the aggregate loading 
in amps on the five South of SONGS lines. 28 The bottom portion of the graphic shows 
all of the phases, while the majority of the graphic shows an expanded view of the phase 
being discussed. The second graphic in each set represents the loading on key facilities 
after each phase. The third graphic in each set shows how power flows redistributed 
through Arizona, Southern California, and Mexico after each phase. Phases 6 and 7 have 
multiple power flow graphics. Phases land 7 include only the second and third type of 
graphics. 



All times are in Pacific Daylight Time (PDT) unless otherwise noted. Times are listed to millisecond (three 
decimal places) or tenth-of-second (decimal place) accuracy when possible. If milliseconds or tenth-of-seconds 
are not listed, the event is reconciled to the nearest second. 

28 

Path 44 flows (complex power in volt amperes, current in amps) were calculated from SONGS PMU data. 
Those readings differ somewhat from disturbance monitoring equipment that was unavailable until completion of 
the inquiry's analysis. The differences are explained by variances in how some minor auxiliary loads are measured 
and in measurement equipment tolerances. 
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The foil owing figure shows all seven phases of the disturbance. 



Figure 4: Seven Phases of the Disturbance 
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Disturbance Phases 



A. Phase i: Pre-Disturbance Conditions 



Phase 1 Summary: 



• Timing: September 8, 2011, before H-NG trips at 15:27:39 

• A hot, shoulder season day with some generation and transmission maintenance outages 

• Relatively high loadi ng on some key faci I ities: H -NG at 78% of its normal rati ng, CV transformers 
at 83% 

• 44 minutes before loss of H-NG, 1 1 D's RTCA results showed that the N-l contingency loss of the 
fi rst CV transformer would result i n an overload of the second transformer above its tri p poi nt 

• An APS technician skipped a critical step in isolating the series capacitor bank at the North Gila 
substation 

September 8, 2011 was a relatively normal, hot day in Arizona, Southern 
California, and Baja California, Mexico, with heavy power imports into Southern 
Cal ifornia from Arizona. I n fact, i mports i nto Southern Cal ifornia were approxi mately 
2,750 MW, just below the import limit of 2,850 MW. September is generally considered 
a "shoulder" season, when demand is lower than peak seasons and generation and 
transmission mai ntenance outages are scheduled. By September 8th, entities 
throughout the WECC region, including some of the affected entities, had begun 
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generation and transmission outages for maintenance purposes. For example, on 
September 8th maintenance outages included over 600 MW of generation in Baja 
California 29 and two 230 kV transmission lines in SDG&E's territory. However, there 
were no major forced outages or major planned transmission outages that would result 
in a reduction of theSOLs in the area. 

■ Pre-Disturbance Conditions in IID 

Despite September being considered a shoulder month, temperatures in 1 1 D's 
service territory reached 115 degrees on September 8th. 30 1 1 D's load headed toward 
near-peak levels of more than 900 MW, which required it to dispatch local combustion 
turbine generation in accordance with established operating procedures. Prior to the 
event, loading on 1 1 D's CV transformers reached approximately 125 megavolt amperes 
(MVA) per transformer, which is approximately 83% of the transformers' normal limit. 
Loading on 1 1 D's Ramon transformer was 153 MVA, which is approximately 68% of its 
normal limit. 

II D'sS Lineties 1 1 D to SDG&E, and through SDG&E, to generation in Mexico at 
La Rosita. It also ties CFE and 1 1 D, through SDG&E's La Rosita international 
transmission line. Before the event, 1 1 D was importing power on theS Line, and thus 
power was flowing northward from thejointly owned I mperial Valley substation to 1 1 D's 
El Centro substation. Flows on the S Line would reverse multiple times during the event. 
When power flowed on theS Linefrom south to north, the implication was that 1 1 D was 
supplied radially through SDG&E. Throughout the event, as power flowed from north to 
south, the implication was that flows intended for SDG&E and/ or CFE were moving 
through II D's system. Eventually, in Phase 6, south to north flows on the S Line would 
activate a RAS that would ultimately trip more than 400 M W of generation at La Rosita 
and the S Line, thereby worsening the loading on Path 44. 

Forty-four minutes prior to the loss of H-NG on September 8, 201X 1 1 D's RTCA 
results showed that the N-l contingency loss of the first CV transformer would result in 
an overload of the second transformer above its tri p poi nt. The 1 1 D operator was not 
actively monitoring the RTCA results and, therefore, was not alerted to the need to take 
any corrective actions. At the time of the event, 1 1 D operators did not keep the RTCA 



The generation was known as Termoelectrica de Mexicali, and will be hereafter referred to as "TDM." It is also 
shown as "TDM" on the Map of Affected Entities. 

^ According to IID, the temperature in El Centro, California reached 115 degrees on September 8, 2011. 
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display visible, and RTCA alarms were not audible. By reducing loading on the CV 
transformers at this pre-event stage, the operator could have mitigated the severe effects 
on the transformers that resulted when H-NG tripped. Si nee the event 1 1 D has required, 
and now requires, its operators to have RTCA results displayed at all times. The loading 
on 1 1 D's CV transformers was pivotal to this event. Loadi ng on the CV transformers is 
influenced by: (1) thepre-contingencyflowon H-NG; (2) load and generation in 1 1 D's 92 
kV network; (3) flow on the S Line; and (4) to a lesser extent, generation connected to 
the Imperial Valley substation. See Figure 5, below. 



Figure 5: Post-Contingency CV Transformer 
Loading Based on All 1 1 D 92 kV Generation 
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■ Pre-Disturbance Conditions in CFE 

At 15:07 CFE's PresidenteJ uarez Unit 11 tripped, which required CFE to activate 
its Baja California Control Area contingency reserves to restore its ACE. At 15:15 PDT 
CFE returned its ACE to where it had been before the unit tripped. Although still 
complying with the spinning reserve requirements, CFE was short on non-spinning 
reserve, with all of its available resources in use or already deployed. 
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■ Pre-Disturbance Focus of WECC RC 

Prior to the event WECC RC operators were monitoring unscheduled flow on 
several paths in Northern California. WECC RCdid not view any of the scheduled 
transmission or generation outages as significant. As illustrated by the chart below, two 
minutes before the event (at 15:25), major paths in the blackout area were operating 
below their Path ratings: 



Major Paths in the Blackout Area 


Established Path 
Ratings/Flow Limits 


Path Loadings 
MW and % 




500kVH-NG 

(Part of Corridor linto blackout area) 


1,800 MW 31 


1,397 MW 
78% 


Path 44 

(Corridor 2 into blackout area) 


2,200 MW 32 


1,302 MW 
59% 


230 kVS Line 

(Part of Corridor 3 into blackout area) 


239 MW 


90 MW 

38% 


SDG&E Import SOL 


2,850 MW 


2,539 MW 
89% 


SDG&E toCFE Path 45 


800MWS-N; 
408MWN-S 


241MWN-S 
60% 



■ Pre-Disturbance Conditions in APS 

APS manages H-NG, a segment of theSWPL. At 13:57:46, the series capacitors 33 
at APS's North Gila substation were automatically bypassed due to phase imbalance 
protection. APS sent a substation technician to perform switching to isolate the 
capacitor bank. The technician was experienced in switching capacitor banks, having 
performed switching approximately a dozen times. APS also had a written switching 
order for the specific H-NG series capacitor bank at North Gila. After the APS system 
operator and the technician verified that they were worki ng from the same switchi ng 
order, the operator read steps 6 through 16 of the switching order to the technician. The 



The limit of H-NG is a portion of the rating of Path 49. The inquiry determined that the limit is approximately 
1,800 MW. 

J With one segment of the SWPL out, the limit increases to 2,500 MW. 

33 A series capacitor is a power system device that is connected in series with a transmission line. It increases the 
transfer capability of the line by reducing the voltage drop across the line and by increasing the reactive power 
injection into the line to compensate for the reactive power consumption. In simple terms, a 50% series 
compensated line means it has the equivalent of 50% of the electric distance (or impedance) of the otherwise 
uncompensated line. 
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technician repeated each step after the operator read it, and the operator verified the 
technician had correctly understood the step. The technician then put a hash mark 
beside each of steps 6 through 16 to indicate that he was to perform those steps. The 
technician did not begin to perform any of steps 6 through 16 until after all steps had 
been verified with the system operator. 

The technician successfully performed step 6, verifying that the capacitor breaker 
was closed, placi ng it i n "local " and taggi ng it out with "do not operate" tags. H owever, 
because he was preoccupied with obtaining assistance from a maintenance crew to hang 
grounds 34 for a later step, he accidentally wrote the time that he had completed step 6 
on the line for step 8. For several minutes, he had multiple conversations about 
obtaining assistance to hang the grounds. Hethen looked back at the switching order to 
see what step should be performed next. His mistake in writing the time for step 6 on 
the line for step 8 caused him to pick up with step 9, rather than step 7. 35 Thus, he 
ski pped two steps, one of them the cruci al step (step 8) of cl osi ng a I i ne switch to pi ace 
H-NG in parallel with the series capacitor bank. This step would bypass the capacitor 
bank, resulti ng i n al most zero voltage across the bank and vi rtual ly zero current through 
the bank. Because he ski pped step 8, when he began to crank open the hand-operated 
disconnect switch to isolate the capacitor bank, it began arcing under load. 36 Hecould 
not manage to toggle the geari ng on the switch to enable its closure, so he stayed under 
the arci ng 500 kV I i ne, determi ned to crank open the switch far enough to break the arc, 
thereby preventing additional damage to the equipment. Figure 6, on the next page, is 
a schematic of the APS series capacitor bank, showing steps seven through nine. 



M Grounds are temporary protective connections that are run from conductive parts of lines, structures, and 
equipment, to earth or some other grounding system that substitutes for earth. If the isolated equipment is 
accidentally energized, grounds are intended to: (1) limit the voltage rise at the worksite to a safe value; and (2) 
provide a pathway for fault current to flow, thereby allowing upstream protective devices to trip. 

3 ^ In human performance analysis, this is known as a "place keeping" error, by failing to physically mark steps as 
they are completed. 

3 ^ An electric arc is a luminous discharge of current that is formed when a strong current jumps a gap in a circuit. 
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Figure 6: APS Series Capacitor Bank 
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B. Phase 2: Trip of the Hassayampa-North Gila 500 kV Line 
Phase 2 Summary: 

• Timing: 15:27:39 to 15:28:16, just before CV transformer No. 2 trips 

• H-NG trips due to fault; APS operators believe they will restore it quickly and tell WECC RC 

• H-NG flow redistributed to Path 44 (84% increase in flow), 1 1 D, and WALC systems 

• CV transformers i mmedi ately overl oaded above thei r rel ay setti ng 

• At end of Phase 2, loading on Path 44 at 5,900 out of 8,000 amps needed to initiate SONGS 
separation scheme 

At 15:27:39, the arc that had developed on each phase of the disconnect switch 
I engthened as the switch conti nued to open, to the poi nt where two phases came i nto 
contact. This caused H-NG to trip to clear this phase-to-phase (A to C) fault. The high- 
speed protection system correctly detected the fault and tripped the line in 2.6 cycles (43 
milliseconds). After discussion with the technician, APS operators erroneously believed 
that they could return the line to service in approximately minutes, even though they had 
no situational awareness of a large phase angle difference caused by the outage. M ore 
time would have been needed to redispatch generation to reduce the phase angle 
difference to the al lowed val ue. APS system operators i nformed CAI SO, Salt River 
Project (SRP), and WECC RC that the line would be reclosed quickly, even though they 
were unaware that this was not possi ble because of the large phase angle difference that 
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existed between H assayampa and North Gila. The inquiry's simulation indicates that the 
post-contingency angular difference was beyond the allowed North Gila synch-check 
relay reclosing anglesetting of 60 degrees, and there would not have been adequate 
generation for redispatch to reduce the phase angle difference to within the allowed 
val ue. APS operators were only able to see the angular difference on E M S displays after 
isolating the North Gila capacitor bank and re-energizing H-NG from the H assayampa 
substation (before closing at North Gila). 

H-NG, which has a flow limit of X800 MW 38 with a 30 minute emergency rating 
of 2,431 M W, was carryi ng \391 M W flowi ng from east to west along the SWPL at the 
ti me of the tri p. As a result of the I i ne tri p, flows redistri buted across the remai ni ng I i nes 
into the San Diego, I mperial Valley, and Yuma areas. The 1 1 D and WALC systems, 
located between the two parallel high voltage Paths, were forced to carry approximately 
23% of the flow that had initially been carried by H-NG. The majority of theflow 
diverted to Path 44, as discussed below. 

I mmediately after the loss of H-NG, the loading on both of 1 1 D's CV transformers 
i ncreased to 130% of thei r normal rati ng and 118.5% of thei r emergency rati ng. The ti me 
overcurrent relays on the CV transformers picked up because the current flow was above 
the overcurrent relay setti ng, and began ti mi ng accordi ng to thei r very i nverse 39 ti me 
delay. TheCV transformers would both trip within 40 seconds of the loss of H-NG. At 
the same time, loading on 1 1 D's Ramon 230/92 kV transformer increased to 94% of its 
normal rating and 85%of its emergency rating. Three seconds after the loss of H-NG, 
SCADA meteri ng for the CV transformer banks stopped recordi ng accurate readi ngs due 
to remote terminal unit (RTU) exceeding maximum scale. 1 1 D and WECC RC no longer 
had accurate information about or situational awareness of the loading on these 
i mportant transformers. 

1 1 D also experienced increased loading on several of its 161 kV lines immediately 
after the loss of H-NG: Blythe-Niland and Knob-Pilot Knob loading increased by 49% 
and 55%, respectively. Flows on 1 1 D's S Line reversed from south to north (SDG&E to 



x See footnote 31, supra. 

" "Very inverse" describes the time/current characteristic of the relays' time delay which is inversely 
proportional to the current magnitude sensed by the relay. That is, the greater the current, the less time before 
the relay will trip. 
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1 1 D) to north to south (I I D to SDG&E) during this phase of the event indicating that 
flows intended for SDG&E were being routed through 1 1 D's 161 and 92 kV systems. 
While 1 1 D was aware of the flow changes on the S Line, it was unable to seethe loss of H- 
NG in real time. 

Flows on WALC's Gila 163/69 kV transformers increased from approximately 12 
MVAto60 MVA, still well below their normal limits of 75 MVA each, but indicative of 
the sudden i ncrease i n f I ows on WALC's system j ust after the I oss of H - N G . WALC al so 
experienced significant voltage drops on its 161 kV system, particularly at Blythe (6.9% 
drop) and Kofa (6.7% drop) substations, due to the increased flows on that system. 

The loss of H-NG interrupted the southern 500 kV path into San Diego. The 
majority of the flow diverted to the northern entry to SDG&E, Path 44. Flow on Path 44 
increased by approximately 84%, from X293 MW to 2,362 MW. This flow equates to a 
ti e current of 5,900 amps rel ati ve to the 8,000 amps requi red to i ni ti ate the SON GS 
separation scheme. 

Because so much of the flow on H-NG was intended for San Diego, the inquiry 
considered whether increasing internal generation in SDG&E 's area would have avoided 
the cascadi ng outages. 40 Figure 7, on the next page, i 1 1 ustrates post-conti ngency 
loading on theCV transformers based on pre-conti ngency loading on H-NG and the 
generation level at 1 1 D's and SDG&E's jointly owned Imperial Valley substation. The red 
area on the graph indicates the large zone in which loading below H-NG's X800 M W 
SOL would load theCV transformers above their trip point. This area demonstrates the 
non-secure N-loperati ng poi nt of the CV transformers. 1 1 shows that the operati ng 
conditions that would reduce the loading on the transformer are: increased generation 
at Imperial Valley, reduced flow on H-NG before it tripped, or both. For example, the 
graph indicatesthatfor the same amount of transfer on H-NG, additional generators 
connected at Imperial Valley would reduce the post-contingency loading on theCV 
transformers. 



w The inquiry's analysis is not intended to suggest specific generation adjustments that could have been made 
by specific entities on September 8, 2011, but rather to show the extent to which the affected entities are 
interdependent. 
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Figure 7 : Post-Contingency CV Transformer 
Loading Based on I mperial Valley Generation 




In general, adding generation in San Diego, CFE, or Imperial Valley and backing 
down generation in APS's system (east of Path 49) would reduce the loading on 1 1 D's 92 
kV system for the loss of H-NG. For example, an additional 600 M W of generation at 
Imperial Valley and a reduction of generation in APS's system by the same amount 
would have reduced the pre-contingency loading on H-NG by 20% and improved the 
post-contingency voltage in WALC's Blythearea by approximately 4%. Under this 
condition, the loading on the CV transformers for the loss of H-NG would be 
approximately 111% of their normal rating (166 MVA), well below their trip setting of 
127%. This is a further demonstration of the importance of including all facilities when 
deriving SOLs. 

After seei ng the alarm for the loss of H - NG, the WECC RC operator promptly 
called the line's operator, APS. APS told WECC RC it could get H-NG restored within 
minutes. While WECC RC was monitoring Rated Paths, it took no action specific to Path 
44, believing it would take five or ten minutes for APS to restore H-NG. As the entire 
event took only 11 minutes, WECC RC did not issue any directives in connection with the 
loss of H-NG. 

Shortly after H-NG tripped, at 15:27:49, one of the combustion turbines at CFE 's 
Central La Rosita substation tripped while producing 156 MW. This trip may have been 
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triggered by transients 41 caused by the initial fault at North Gila and subsequent trip of 
H-NG. Loss of this unit further increased the flow on Path 44, raising the current to 
6,200 amps out of the 8,000 needed to initiate the SONGS separation scheme. 
However, the La Rosita trip alone was not significant in causing the cascading that 
followed. 42 CFE was also unaware in real time that H-NG had tripped. After losing the 
Central La Rosita unit, CFE was unable to recover its ACE with its own resources, and at 
15:30, it requested 158 M W of emergency assistance from CAI SO for the remainder of 
the hour. 



Phase 2 Graphics 



South of SONGS - Calculated Phase Current 




4 ^See footnote 21. CFE stated that the trip was triggered by transients. 

4 ^ The Modeling and Simulation team conducted a "what if" simulation and determined that, even without the 
inadvertent tripping of 160 MW of generation at La Rosita, the overloads and ensuing blackout would still have 
occurred. 
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C. Phase 3: Trip of the Coachella Valley 230/92 kV Transformer and 
Voltage Depression 

Phase 3 Summary: 



• Timing: 15:28:16, when CV transformer bank No. 2 tripped, to just before 15:32:10, when Ramon 
transformer tripped 

• Both CV transformers tripped within 40 seconds of H-NG tripping 

• 1 1 D knew losing both CV transformers would overload Ramon transformer and S Line connecting it 
with SDG&E 

• Severe low voltage i n WALC's 161 kV system 

• At end of Phase 3, loading on Path 44 at 6,700 amps out of 8,000 needed to initiate SONGS 
separation scheme 

At 15:28:16, less than a minute after H-NG tripped, 1 1 D's CV transformer bank 
No. 2 tripped on the 230 kV side. The CV overload protection relays detected an 
overload i mmediately after H-NG was lost. The overloads were caused by through-flows 
on 1 1 D's 92 and 161 kV systems which parallel APS's 500 kV system. The normal ratings 
for these transformers are 150 MVA, but immediately after H-NG tripped, each CV 
transformer was carryi ng more than 191 M VA. The rel ays were set to tri p at 
approximately 127% 43 of the transformers' normal ratings, or 1912 MVA at nominal 
voltage. The i n verse ti me relays took 37.5 seconds to tri p bank No. 2 and 38.2 seconds to 
trip bank No. 1 Thus, CV bank No. 1 tripped only 677 milliseconds after bank No. 2, 
again on the 230 kV side. Although the primary winding or high side voltages of the CV 
transformers are 230 kV, the banks were not considered as elements of the BES because 
their secondary winding or low side voltages are below 100 kV. As discussed in detail in 
Section IV, because these transformers and the underlying 92 kV system were not 
classified as elements of the BES, 1 1 D, neighboringTOPs, and WECC RC did not assess 
theimpact of critical external contingencies on overloading the CV banks, the effect of 
I osi ng the CV banks and the subsequent i mpact on the Ramon bank, and, f i nal ly thei r 
overal I adverse effect on BPS rel iabi I ity. 

1 1 D was aware of the potential for local cascadi ng if the CV transformers tri pped. 
1 1 D's next-day plan for September 8, 2011 which was not based on updated studies, 
indicated that if both CV transformers tripped, 44 the Ramon 230/92 kV transformer 
would trip and theS Line tie with SDG&E would overload to 109% of its normal rating. 
The next-day plan also indicated that this overloading, in turn, would result in tripping 



4 ^ IID's transformer protection philosophy specifies trip settings at 120% of normal ratings. IID chose the closest 
available relay tap, which was approximately 127% of the normal rating. 

44 This contingency scenario had nothing to do with H-NG tripping. IID's studies did not show any effect on the 
CV banks resulting from the loss of H-NG. 
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generation because the S LineRAS trips generation supplied to Imperial Valley when the 
S Line loads to 108% of its normal rating. 1 1 D's next-day mitigation plan for loss of the 
CV transformers requi red starti ng turbi nes at Coachel la and N i land and aski ng CAI SO to 
redispatch generation to relievetheS Line. This was a post-contingency mitigation plan. 
But after the event 1 1 D's operator admitted that if the CV transformers tripped on 
overload, he would have "very little time to mitigate the Ramon [transformer], if at all." 
Even the quickest-starting turbi nes take about 10 minutes to start and ramp to full load, 
but 1 1 D effectively had only four minutes before the Ramon transformer would trip, after 
the loss of the CV transformers. 

The loss of the CV banks caused flows on the S Li ne between SDG&E and 1 1 D to 
again reverse direction. Because its load exceeded internal generation, 1 1 D began pulling 
power from SCE through SDG&E due to the loss of key facilities in 1 1 D's northern 
system. The tripping of the second CV bank also open-ended the Coachel I a Valley- 
Ramon 230 kV "KS" Line(at CV), which was carrying about 41 MVA. Thisfurther 
increased loading on the Mi rage- Ramon 230 kV line and through-flow from 1 1 D's 230 kV 
col lector system through Devers, but had little effect on the overal I disturbance. By 
15:31:35, 1 ID's operators switched in 92 kV capacitor banks at Avenue 42, Avenue 58, 
and Highline due to low voltage. 

The loss of 1 1 D's two CV transformers caused the aggregate current on Path 44 to 
increase from 6,200 amps to 6,600 of the 8,000 amps necessary to trigger the SONGS 
separation scheme. H owever, by the end of this Phase aggregate Path 44 current 
reached 6,700 amps. 

The loss of the CV banks caused a severe voltage depression on the WALC 161 kV 
system south of Blythe. During this period, loads in that area (largely irrigation pumps) 
were highly susceptible to motor stalling, which can create additional reactive demand 
and exacerbate transmission loading, both of which contribute to additional voltage 
decline. See Figure 8, on the next page. At 15:28:18, the Blythe 161 kV bus alarmed at 
142 kV (0.882 per unit). 45 WALC continued to experience severe low voltage on its 161 
kV system until the S Line tripped at 15:38:02.4. 



Other alarms and low voltage readings followed throughout WALCs system one to nine seconds later, 
including the Parker-Kofa 161 kV line, which alarmed for overload at 169 MVA (167 MVA rating); Kofa 161 kV bus 
voltage recorded at 143 kV (0.888 per unit); Knob 161 kV bus voltage recorded at 142 kV (0.882 per unit); Parker 
161 kV bus voltage recorded at 149 kV (0.925 per unit); Gila and Goldmine 161 kV bus voltages recorded at 144 kV 
(0.894 per unit); and Parker 230 kV bus voltage recorded at 222 (0.965 per unit). 
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Figure 8: Blythe 161kVVoltage 




On September 8, 2011 CAI SO had partial visibility of 1 1 D's system, but could not 
see that the CV banks had tri pped. Prior to the event CAI SO and 1 1 D had been worki ng 
together to increase their mutual visibility and those efforts are continuing. Currently, 
CAI SO receives loadi ng data from the 230 kV side of the CV transformers. 

Despite the fact that it did not consider the CV banks to be part of the BE S, 
WECC RC does observe much of 1 1 D's 92 kV system i n real ti me, i ncl udi ng the CV banks. 
TheWECC RC operator did notice the CV transformers trip, but he was focused on when 
APS would return H-NG to service. 
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Phase 3 Graphics 



South of SONGS - Calculated Phase Current 




15:28:17- Two Coachel I a Valley 230/92 kV transformers 
and the Coach el I a Valley Ramon 230 kV "KS" line tripped. 
(030) 
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35:28:18 




D. Phase 4: Trip of Ramon 230/92 kV Transformer and Collapse of IID's 
Northern 92 kV System 



Phase 4 Summary: 



• Timing: 15:32:10 to just before 15:35:40 

• 1 1 D's Ramon 230/92 transformer tripped at 15:32:10, was set for 207% of its normal rating instead 
of its design setting of 120%, which allowed it to last approximately four minutes longer than CV 
transformers 

• 1 1 D experienced undervoltage load shedding, generation and transmission line loss in its 92 kV 
system 

• Path 44 loadi ng i ncreased from approxi mately 6,700 amps, to as high as 7,800 amps, and ended at 
around 7,200 amps (out of 8,000 needed to initiate the SONGS separation scheme) 

At 15:32:10.621 less than five minutes after the trip of H-NG, 1 1 D's Ramon 
230/92 kV transformer tripped on the 92 kV side. The normal rating for this 
transformer was 225 M VA, and its relays were set to trip above 207% of its normal 
rating, or 466 M VA. Before it tripped, the SCADA metering for the Ramon bank had 
stopped recordi ng accurate readi ngs due to RTUs exceedi ng maxi mum scale, j ust as for 
theCV banks. Following the loss of the CV transformers, the inverse time relays took 
less than four mi nutes to tri p the Ramon transformer. 1 1 D had i ntended to set the 
Ramon transformer to trip at 120% of its normal rating. Had it been set at this level, the 
Ramon transformer would have tripped almost immediately after the loss of the CV 
transformers, approxi mately four mi nutes earlier than the time of its actual trip. 1 1 D 
believed that the Ramon transformer would overload beyond the trip point upon the loss 
of both CV transformers. I ts next-day plan noted, "the Ramon Bank #1 transformer wi 1 1 
overl oad and rel ay out of servi ce because the overcurrent setti ngs are set to tri p at 
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120%." 1 1 D's next-day plan relied on a post-conti ngency operati ng phi losophy of starti ng 
the Coachel la Gas Turbi nes to mitigate overloads fol lowi ng the loss of the CV 
transformers, but the plan was unrealistic as 1 1 D would not have had time to start any 
additional generation between the loss of the CV transformer banks and the loss of the 
Ramon transformer. 

Within less than one second after the loss of the Ramon transformer, automatic 
distribution undervoltage protection in 1 1 D's northern 92 kV system began tripping 
distribution feeders and shedding load. From 15:32:11 to 15:33:46, 444 MW of 1 1 D's load 
tripped, with nearly half of the load being shed within 10 seconds of the Ramon 
transformer tripping. As illustrated in Figure 9, below, the severe voltage depression 
fol lowi ng the loss of the Ramon transformer appears to have prompted a local voltage 
collapse within 1 1 D's northern 92 kV system, evidenced by both the steep drop-off in 
voltage as well asasharp risein reactive power flow due to motor stalling. 



Figure 9: 92kV Voltage (per unit) at Avenue 58 
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The loss of 1 1 D's northern resources and subsequent system response caused 1 1 D 
to lose multiple generators connected to its 92 kV system, including 1 1 D's Ni land Gas 
Turbine 2 (generating 45 M W), 1 1 D's CV Gas Turbine 4 (generating 20 MW), 
independent power producer Col mac's unit (generating 46 MW), and 1 1 D's Drop 4 Unit 2 
Hydro Generator (generating 10.3 M W). 

II D also began losing transmission lines. The Blythe-Ni land 161 kV "F" Line, 
which saw increased loading during Phase 2, tripped at 15:32:13 (approximately 3 
seconds after I oss of the Ramon banks) . I ts normal rati ng was 165 M VA, and it was set to 
trip at 129% of the normal rating (212 MVA at nominal voltage) with a 3-second time 
delay. 46 TheNiland-CV 161kV "N" Line, carrying 83 MVA, tripped approximately 2 
seconds later at 15:32:15.29 due to Zone 3 distance protection. 47 

I n WALC's territory, the Blythe-Goldmine-Knob and Parker-Kofa 161 kV lines 
overloaded approximately four seconds after the Ramon transformer tripped, at 
15:32:14, but did not trip. These lines each had a normal rating of 167 MVA, but were 
loaded to 177 MVA. Power flows redistributed through the Parker and Blythe areas after 
II D lost the Blythe-Ni land line. WALC took some actions in an attempt to arrest the 
voltage depression it was experiencing, including a directive to start hydropower 
generation units Parker 3 and 4 for voltage support at 15:34:07. Atthetime, Parker area 
voltage was at 150 kV (0.932 per unit). WALC also switched in shunt capacitors on the 
69 kV system at Gila and Kofa. Atthetime, voltage at Gila was at 65.5 kV (0.906 per 
unit) while Kofa was at 59 kV (0.86 per unit). 

CAI SO attempted to bring on generation through its exceptional dispatch 48 
process to bring Path 44 back within its limit of 2,500 MW, anticipating that it had 30 
minutes to do so. At 15:35, it dispatched the Larkspur No. 2 peaking unit (rated 50 M W) 
within San Diego, which has a 20-minute start-up time. Also at this time, APS began 
taking steps to restore H-NG by completing the bypass of the series capacitor bank. 



a " Based on the last available SCADA scan before the line tripped, the voltage at Blythe was at 123.1 kV (0.765 per 
unit) and the line was loaded to 172 MVA. Based on these measurements, the line was carrying 807 amps at the 
last time recorded; the relay was set to trip with a 3-second time delay at 762 amps. 

^ A distance relay is a relay that compares observed voltage and current on a line and operates when that ratio is 
below its preset value. Zone 3 relays are typically set to protect against faults that are more than one substation 
away from the observed line as backup protection. An appropriate time delay should be set in the relay to give 
the remote station relays the opportunity to operate and isolate the minimum amount of equipment necessary to 
clear the fault. A common issue with the application of Zone 3 relays is that they can restrict the loading on 
transmission lines (e.g. the N Line) during abnormal system conditions like those present on September 8th. 

CAISO's exceptional dispatch process involves calling on generators outside of the market automated 
dispatch process. 
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During Phase 4, aggregate loading on the South of SONGS 230 kV transmission 
lines increased from approximately 6, 700 amps to as high as 7,800 amps. The loading 
settled around 7,200 amps and remained therefor the rest of Phase 4. 

Phase 4 Graphics 



South of SONGS - Calculated Phase Current 
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Time: 15:32:10 - The Ramon 230/92 kV transformer 
tripped and 1 1 D shed 444 MW of load. (110) 




-43- 



FERC/NERC Staff Report on the September 8, 201 1 Blackout 



Time: 15:32:35 




E. Phase 5: Yuma Load Pocket Separates from II D and WALC 
Phase 5 Summary: 



• Timing: 15:35:40 to just before 15:37:55 

• The Gila and Yucca transformers tripped, isolating the Yuma load pocket to a single tie with SDG&E 

• Path 44 loadi ng i ncreased from 7,200 to 7,400 amps after Gila transformer tri pped, and ended at 
7,800 amps after loss of the Yucca transformers and YCA generator (very close to the 8,000 amps 
needed to initiate the SONGS separation scheme) 

At 15:35:40, approximately eight minutes after H-NG tripped, WALCsGila 
163/69 kV transformers tripped due to time-overcurrent protection. The two 
transformers are each rated 75 M VA, but the 69 kV bus section that connects the 
transformers to the rest of the 69 kV substation is rated 3,200 amps (143 M VA at 
nominal voltage), and theovercurrent protection is set accordingly at 3,200 amps. The 
bus was carryi ng 3,332 amps at the ti me of the tri p. 

One minute later, at 15:36:40, the Yucca 163/69 kV transformers land 2 tripped 
when their common 69 kV breaker tripped due to overload protection. Bank No. lis 
owned by 1 1 D and is rated 73 M VA, and bank No. 2 is owned by APS and is rated 75 
M VA. The 1 1 D Yucca generator and four out of the six APS combustion turbines 
connected to APS's 69 kV system were off I i ne at the ti me of the event, as was the 1 1 D 
GT21 combustion turbine on the 161KV side. These generators may have supported load 
i n the area had they been i n service. Al most i mmediately, the Pi lot Knob breaker on the 
Pilot Knob-Yucca 161 kV "AX "transmission line, which is effectively the 161 kV breaker 
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for the Yucca 163/69 kV transformers, received a direct transfer trip from the Yucca 
transformer overload protection, thereby tripping the AX Line. As a result of the loss of 
the Yucca and Gila transformers, the Yuma load pocket was isolated to only one tieto the 
SDG&E system, causing loading on each N. Gila 500/69 kV transformer bank to increase 
from 57MVAtol64MVA. 

Less than one second after the Yucca transformers and AX Line tripped, at 
15:35:40, the Yuma Cogeneration Associates (YCA) combined cycle plant on the Yuma 
69 kV system tripped. The combustion turbine is rated at 35 M W and the heat recovery 
unit is rated at 17 MW, totaling 52 MW. 1 1 appears that both units were fully loaded at 
the time of the trip. The cause of the trip isunknown, but the loss of the YCA unit 
hastened the col I apse of the Yuma load pocket. 

Approximately one minute later, at 15:37:41, a common 161 kV breaker tripped 
1 1 D's Pilot Knob 163/92 kV transformers Nos. 2 and 5 for No. 2 overload protection. The 
overload protection was set to trip the banks at 121% of the normal rating (37.5 MVA at 
nominal voltage). 

At WALC's request, between 15:36:48 and 15:36:52, SCE directed Metropolitan 
Water District operators to drop 80 M W of pumping load attached to the Gene 
substation (near Parker) to improve 230 kV voltage support at Parker in an attempt to 
arrest decl i n i ng voltages. 

As it had done during Phase 4, CAI SO ordered exceptional dispatch to bring Path 
44 below its 2,500 M W limit. At 15:36:00, CAI SO called SCE and ordered an exceptional 
dispatch of Larkspur Peaking Unit No. 1 (rated 50 MW), and Kearny GT2 and GT3 (each 
rated 59 MW) to go to full load. The Larkspur unit takes 20 minutes to start, and the 
Kearny units are 10-minute "quick start" peaking generators. All of these units were 
off I i ne at the ti me, and they were unable to come on I i ne before the system col lapsed. 49 

The tri ppi ng of the Gi la 161/ 69 kV transformers caused the aggregate loadi ng on 
Path 44 to increase from approximately 7,200 amps to approximately 7,400 amps, out of 
the 8,000 amps necessary to initiate the SONGS separation scheme. After the loss of the 
Yucca 161/69 kV transformers, the YCA plant, and the Pilot Knob 161/92 kV 
transformers, the loading further increased to approximately 7,800 amps. 



^ Larkspur generation is connected to the SDG&E 69 kV system south of Otay Mesa, and Kearny generation is 
connected to the SDG&E 69 kV system in northern San Diego. 
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Phase 5 Graphics 
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Time: 35:35:31- YumaCogen (YCA) tripped. (180) 
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Time: 35:37:42 




F. Phase 6: High-Speed Cascade, Operation of the SONGS Separation 
Scheme and Islanding of San Diego, IID, CFE, and Yuma 

Phase 6 summary: 



• Timing: 15:37:55 to 15:38:212 

• IID'sEI Centro-Pi lot Knob line tripped, forcing all of I ID'S southern 92 kV system to draw from 
SDG&E via the S Line 

• S Line RAS operates, tripping generation at Imperial Valley and worsening the loading on Path 44 

• S Line RAS trips S Line, isolating 1 1 D from SDG&E 

• Path 44 exceeds trip point of 8,000 amps, to as high as 9,500 amps 

• SONGS separation scheme operates and creates SDG&E/ CFE/Yuma island 

When the El Centro-Pilot Knob 161kV line tripped at 15:37:55 (10 minutes after 
loss of H-NG), it isolated the southern 1 1 D 92 kV system onto a single transmission line 
from SDG&E: the S Line. Forcing all of the remaining load in IID to draw through the 
SDG&E system pushed the aggregate current on Path 44 to 8,400 amps, well above the 
tri p poi nt of 8,000 amps. I f the aggregate current on Path 44 remai ned above 8,000 
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amps, the definite minimum time relay 50 would initiate the SONGS separation scheme 
to separate SDG&E from SCE at SONGS. 

1 1 D's El Centro-Pilot Knob 161 kV line open-ended at El Centro when a 161 kV 
breaker at El Centro tripped on Zone 3 relay protection 51 with a one second delay. The 
apparent i mpedance detected on the Zone 3 relay at E I Centro was hoveri ng near its tri p 
zone immediately foil owing the Pi lot Knob 163/92 kV transformer trips (12 seconds 
earlier), but did not cross into the Zone 3 tripping region until this time. 

By this ti me i n the event, the South of SONGS I i nes were San Diego's only source 
of critical imported generation, and were also keeping 1 1 D and CFE's Baja California 
Control Area from goi ng dark. I f the aggregate current was brought below 8,000 amps, 
the blackout could have been avoided, but at this point no operator action could have 
occurred quickly enough to save the South of SONGS Path. H ad there been formal 
operating procedures that recognized the need to promptly shed load as the aggregate 
current approached 8,000, and had operators been trained on the 8,000 amp set point, 
it is possible that operation of the SONGS separation scheme could have been averted by 
earlier control actions. 

M i 1 1 i seconds after the loss of 1 1 D's E I Centro- Pi lot Knob 161 kV I i ne, at 
15:37:55.890, NextEra's Buck Boulevard combustion turbine generator tripped due to 
operation of SCE's Blythe Energy RAS, dropping 128 M W of generation. 52 This was 
caused by a reduction of counter-flows on thej ulian Hinds-Mi rage 230 kV line that had 
been created by heavy flows from the J ulian Hinds- Eagle Mountain 230 kV linefeeding 
toward the WALC 161 kV system to support the heavy north to south 161 kV flows toward 
Pilot Knob. When the El Centro-Pilot Knob 161 kV line tripped, those counter-flows 
disappeared, initiating the RAS operation. The Buck Boulevard heat recovery unit 
ramped down by 82 M W over the next few minutes. The Buck Boulevard combined cycle 
plant was generating 409 MW (535 MW rating) at the time the combustion turbine 
tripped. Tripping the Buck Boulevard generator did not increase loading on Path 44, 
because it is not located south of Path 44. 



A definite minimum time relay can operate in one of two ways. When current reaches a certain value, the 
relay will operate with a definite time delay that reflects the relay's fastest operating time. Before the relay 
reaches that value, the time for the relay to operate is inversely proportional to its observed current magnitude. 
During the event, the relay operated while following the latter characteristic. 

^See footnote 47, supra. 

M The Blythe Energy RAS, among other functions, trips generation owned by NextEra to protect the Julian Hinds- 
Mirage 230 kV line from overloading with east to west flows for a potential loss of the Julian Hinds-Eagle 
Mountain 230 kV line. Buck Boulevard is connected to SCE's 230 kV system in the Blythe area. 
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J ust three seconds after the loss of 1 1 D's El Centra- Pi lot Knob 161 kV I i ne, at 
15:37.58.2, the S Line RAS at I mperial Valley Substation initiated the tripping of two 
combined cycle generators at Central La Rosita in Mexico. The S Line RAS currently 
protects El Centra's 163/92 kV transformer No. 2 by initially tripping a combination of 
CLR 1 1 generators when the flow on the S Line exceeds 269 MW flowing northward from 
SDG&E into 1 1 D. Two combustion turbines were loaded to 152 MW (193.5 M W rating), 
and 153 MW (193.5 M W rating), respectively, and the associated steam heat recovery 
unit (which also tripped following loss of the turbines) was loaded to 127 MW (159.3 
rating), totaling 432 MW of generation. 

Loss of the CLR II generation drove the South of SONGS flows from about 8,400 
amps to about 9,500 amps, which remained above the8,000 amp setting of the SONGS 
separation scheme. The inquiry's simulation showed that had the S Line tripped without 
the S Line RAS tripping the CLR II generation, the flow on Path 44 would have fallen 
below 8,000 amps to settle at an estimated 7,730 amps, and the SONGS separation 
scheme might not have operated. 53 

Approxi mately four seconds after the S Li ne RAS tri pped the CLR 1 1 generators, 
at 15:38:02.4 the S Line RAS tripped the S Line itself due to flow above 289 MW toward 
1 1 D from SDG&E. Tripping of this line created an 1 1 D island. 1 1 D reported that from 
15:37:59 to 15:40:24, 507.85 M W of load tripped on its system, mostly in the southern 92 
kV system. 

The tripping of the S Line meant that IID was no longer pulling power from 
SDG&E and CFE through Path 44, so the aggregate Path 44 flows decreased from 
approximately 9,500 amps to approximately 8,700 amps, but were still above the 8,000 
amps required to trigger the SONGS separation scheme. 

At 15:38:212, not quite 11 minutes after H-NG tripped, the SONGS separation 
scheme operated, reconfiguring the SONGS 230 kV switchyard and isolating the SONGS 
generators onto the SCE system to the north. This reconfiguration effectively separated 



M The inquiry's simulation showed that if the S Line RAS tripped only the S Line, 1 1 D's system would still have 
collapsed, but San Diego and the Yuma load pocket would likely have survived. Voltages would have remained 
acceptable, and the 230 kV system around SONGS may have experienced minor overloads. While this would have 
resulted in a large phase angle difference on H-NG, the fact that the SONGS separation scheme would not have 
operated would have allowed time for system operators to make the load and generation changes necessary to 
reduce the phase angle difference. 

Had the S Line RAS not operated at all, or only operated to trip the CLR II generators, Path 44 flows would have 
settled above the 8,000 amp threshold and thus the SONGS separation scheme would still have operated. 



-49- 



FERC/NERC Staff Report on the September 8, 2011 Blackout 



all five South of SONGS 230 kV transmission lines from the SONGS units and theSCE 
system, and separated SDG&E from the rest of the Western I nterconnection. Operation 
of the SONGS separation scheme created an island consisting of the SDG&E system, the 
remaining Yuma-area load connected through the 500 kV system from M iguel to North 
Gila, and CFE's Baja California Control Area. 

September 8, 2011 was the first time that the SONGS separation scheme had 
ever activated, and its effects on neighboring systems had not been studied. Although 
this sequence of events has focused on how the loss of elements combi ned over the 11 
minutes to exceed the 8,000 amp SONGS separation scheme trigger, in real time, no 
entity was monitori ng that I i mit or recognized the potential consequences of its 
operation. 

WECC RC, responsible for the reliable operation of the BPS, and with having a 
wide area view of the BPS, did not have any alarm that would alert operators before 
operation of the separation scheme. Although WECC RC operators were monitoring the 
Path I i mit on Path 44, they were not watch i ng the aggregate flows with respect to the 
SONGS separation scheme trigger. WECC RC operators noticed thefive South of 
SONGS breakers open after the scheme had already operated. 

CAISO, the TOP for SDG&E and SCE, did not have any alarms specifically tied to 
the operation of the SONGS separation scheme either. CAI SO only has alarms for when 
Path 44 exceeds its Path rating, but had no ability to monitor the SONGS separation 
scheme, set at 3,100 M W (8,000 amps). After the loss of H-NG, which caused Path 44 to 
exceed its Path rating, CAISO operators were primarily concerned with returning flows 
on Path 44 to below the Path rati ng of 2,500 M W, but bel ieved they had 30 mi nutes to 
do so. U nl i ke Path rati ngs, the separation scheme would not al low CAI SO operators 30 
minutes to reduce flows on Path 44. CAI SO did attempt to dispatch additional 
generation within SDG&E to reduce flows on Path 44. The other method to reduce flows 
would have been to manually shed load in SDG&E in time to prevent operation of the 
SONGS separation scheme. SDG&E esti mates that it could have shed approxi mately 240 
M W in between two and two-and-a-half minutes. However, SDG&E was never 
instructed to shed load and was unaware of the need to shed load. 
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South of SONGS - Calculated Phase Current 



South of SONGS - Calculated Phase Current 
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G. Phase 7: Collapse of the San Diego/CFE/Yuma Island 
Phase 7 Summary: 



• Timing: J ust after 15:38:212 to 15:38:38 

• Underfrequency Load Shedding (UFLS) was not able to prevent the SDG&E/CFE/ Yuma island 
from col lapsing 

• SONGS nuclear units shut down even though they remai ned connected to the SCE side of the 
SONGS separation scheme 

Duri ng phase 7 of the event the SDG&E/ CF E/ Yuma i sland broke i nto three 
separate islands, all of which collapsed due to an imbalance between generation and 
demand, resulting in severe underfrequency which tripped both loads and generation. 

The SDG&E/CFE/ Yuma island created by operation of the SONGS separation 
scheme had a significant imbalance between generation and load from the beginning. As 
a result the frequency in the island rapidly declined. By less than a second after the 
SONGS separation scheme activated (15:38:22), the UFLS programs of SDG&E, APS, 
and CFE had all began activating within the island. Figures 10 and 11 below show the 
frequency within the island as it collapses. All steps of the UFLS systems activated and 
system frequency in the island briefly stalled at approximately 57.2 hertz(Hz). CFE's 
UFLS analysis showed 512 M W of load shed by 15:38:21901 

H owever, the same analysi s showed that three CF E generators, total i ng 459 M W, 
tripped offline beginning at 15:38:21905, partially negating CFE's UFLS actions. In 
addition, a number of smaller generators, totaling about 130 MW, tripped only 0.5 
seconds later while CFE was still connected to SDG&E and whi I e SDG&E 's UFLS 
program was still workingtoshed load. 54 SeeFigurell below. The net effect of CFE's 
UFLS acti ons and generator tri ps— 512 M W shed by U F LS and 590 M W of tri pped 
generati on— was that CF E 's i mports from SDG&E i ncreased from approxi mately 440 
M W to approximately 520 M W. This worsened CFE's system conditions and increased 
the stress on SDG&E before SDG&E 's underfrequency separation protection systems 
opened the ties between CFE and SDG&E. SDG&E also had three generators with 
underfrequency protection that operated at 57.3 Hz, above the frequency at which the 
system leveled out. Due to these early generation losses, the frequency conti nued to 
decl i ne below 57 H z, which was the underfrequency setti ng for the majority of generators 
in theisland. Thus, theisland blacked out, shortly after separating into three sub- 
islands. 



3 The fact that several generators tripped during load shedding suggests that CFE may benefit from analyzing 
whether its UFLS program and generator underfrequency protection systems are coordinated. 
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Figure 10: Frequency, Voltage in the SDG&E/Yuma/CFE Island 




Figure LL Frequency Performance in the SDG&E/Yuma/CFE Island 
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TheCFE island separated from SDG&E after their only two remaining ties 
tripped in rapid succession. At 15:38:22.2, the Otay Mesa-Tijuana 230 kV transmission 
line open-ended at Tijuana in CFE's territory due to underfrequency protection. 55 Less 
than a second later, at 15:38:23.13, the I mperial Valley-La Rosita 230 kV transmission 
line open-ended at I mperial Valley in SDG&E 's territory by underfrequency 
protection. 56 According to CFE, its UFLS program was not designed for the operation of 
a SDG&E/ CFE/ Yuma island, but for the operation of a "southern WECC island." 

The Yuma island separated from SDG&E at 15:38:23.12, when the I mperial 
Valley-North Gila 500 kV transmission line tripped by underfrequency protection. APS's 
UFLS operated on 26 out of the 28 feeders i n the Yuma area prior to the loss of the local 
Yucca steam generators that were on I i ne. H owever, there was i nsuff i ci ent I ocal 
generation to stabilize the load pocket in Yuma. At 15:38:38, the Yuma island internal 
units tripped on underfrequency protection. 

At about the same ti me that it separated from CF E and APS's Yuma pocket, 
SDG&E lost four generati ng units, total i ng 570 M W, due to the generators' 
underfrequency protection. 57 

Although the SONGS generators remained connected to the SCE side of the 
switchyard at SONGS, at about 15:38:27.5, or approximately six seconds after the 
SONGS separation scheme initiated, the SONGS turbines both experienced a brief 
acceleration in speed and tripped due to turbine control logic. At the same time, local 
system frequency at SON GS was observed to spi ke from 59.974 H z to 61203 H z. After 
theinitial impulse caused by the system separation, the frequency in the main body of 
the Western I nterconnection peaked near 60.170 Hz. This can be seen on Figures 12 and 
13, on the next page. The turbine trip initiated a reactor shutdown, and the units began 
coasting down. A little more than a second later, at 15:38:28.963, SONGS Unit 3 
electrical ly disconnected from the system, and less than three seconds after the reactors 



33 The Tijuana end opened instantaneously. Subsequently, the Otay Mesa end of the line in SDG&E's territory 
opened at 15:38:23.044 by underfrequency protection (with 1-second delay). 

JO The line's underfrequency setting was 57.9 Hz, with 1-second delay. The instantaneous underfrequency 
protection scheme at La Rosita in CFE's territory failed to operate due to a bad fuse connection. 

^ At 15:38:23.000, the Palomar Energy Center combustion turbines CT1 and CT2 tripped on underfrequency, 
followed by the heat recovery unit ST at 15:38:23.07 (all set to trip at 57.3 Hz with a 750 millisecond time delay). 
CT1 was loaded to 160 MW, CT2 was loaded to 165 MW, and ST was loaded to 195 MW at the time of the trips. It is 
believed that additional unit Goal Line LP, generating 50 MW, tripped around the same time due to a 58 Hz 
frequency with a 1-second time delay. 
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shut down, at 15:38:30.209, SONGS Unit 2 electrically disconnected from the system. 
Loss of the 2,300 M W of SONGS' generation effectively reduced the loss of load for the 
main body of the Western I nterconnection from a 3,400 MW loss to a net X100 MW load 
loss. This made the recovery from the resulti ng overfrequency event much easier. The 
SONGS generators did not loseoffsite power because the SONGS switchyard was still 
connected to the SCE system. 



Figure 12 : Frequency Excursion in WECC 

I nterconnection I mmediately after the SONGS 




Figure 13: SONGS Generation Trips and 
Auxiliary Loads Transfer to 230 kV Bus 
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By 15:38:38, theSDG&E, CFE and Yuma islands had all collapsed, leaving 
approxi mately 2.7 mi 1 1 ion customers without power. 



Phase 7 Graphics 



Time: 15:38:30 - The South of SONGS Separation Scheme 
operates and both SONGS units tripped. (300) 
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Yuma Separates (Time: 35.38.23.32) 




CFE Separates (Time: 35.38.23.33) 
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SDG&E, CFE, and Yuma Blackout ( by 15.38.30) 
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H. System Restoration 



None of the affected entities needed to implement black start plans because they 
al I were abl e to access sources of power from thei r own or a nei ghbor's system that was 
still energized. The restoration process generally proceeded as expected, and some 
entities restored load more quickly than they had expected. The following charts 
indicate how long it took the affected entities to fully restore their lost load, generation, 
and transmission. 



LOAD RESTORATION EFFORTS 



1 



Entities 


Demand 
1 nterrupted 
(MW) 


Time Until 
Demand Fully 
Restored 


Date 
Restored 


100% 
Demand 
Restored 
(hrs) 


Number of 
Customers 
Affected 














SDG&E 


4,293 


03:23 


9/9/11 


12 


L4 Million 


CFE 


2,205 


0137 


9/9/11 


10 


llMillion 


IID 


929 


2140 


9/8/11 


6 


146,000 


APS 


389 


2112 


9/8/11 


6 


70,000 


WALC 


74 


22:23 


9/8/11 


6.5 


5 58 



DO The majority of WALCs lost load (64 MW) affected APS customers. SCE lost 117 fringe load customers who 
were served by the SDG&E system. 
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GENERATION RESTORATION EFFORTS 


Entities 


Generation 

1 n<t (W\\M) 

l_LOL V 1 1 V V J 


Time 

Generation 
Restored 


Date Restored 


Generation 












SCE 


2,428 


06:33 


91 ill u. 


87 


SDG&E 59 


2,229 


06:20 


9/10/11 


39 


CFE 


1,915 


23:43 


9/10/U 


56 


IID 


333 


20:42 


9/8/11 


5 


APS 


76 


20:37 


9/ 8/ 11 


5 



TRANSMISSION RESTORATION EFFORTS 


Entities 


Final 

Transmission 
Restored (kV) 


Time 

Transmission 
Restored 


Date 
Restored 


Transmission 
Restored (hrs) 


IID 


230 


03:37 


9/9/11 


12 




161 


00:31 


9/9/11 


9 


SDG&E 


500 


17:36 


9/8/11 


2 




230 


03:47 


9/9/11 


12 


APS 


500 


16:51 


9/8/11 


15 


WALC 


161 


17:09 


9/8/11 


1560 


CFE 


230 


04:03 


9/9/11 


12.5 




115 


0158 


9/9/11 


10 



J=7 According to SDG&E, after restoring the SDG&E transmission systems, GAISO took over restoring SDG&E's 
generation. 

^ This represents the time it took WALC to restore its 161/69 kV Gila transformers, however, none of WALC's 
transmission lines were lost in the outage. 
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WECC RC could have taken a more active role in coordinating the restoration 
efforts. WECC RC hasthelargest area of visibility and more advanced real -time study 
tools than theTOPs. During a multi-system restoration, issues are likely to arise 
between neighboring BAs and TOPsthat may require either a neutral decision maker, or 
rapid technical analysis of unplanned system conditions. WECC RC is uniquely situated 
to provide such assistance. WECC RC should clarify its role, and the real-time 
information it can provide, in emergency situations I ike a multi-system restoration. 
WECC RC should also specifically address the issue of coordination among other 
functional entities (like BAs andTOPs) in its operating area, outlining the areas of 
responsibility during system restoration and other emergencies. 

The inquiry reviewed recordings and other data about restoration which 
disclosed the foil owing incidents that could have benefitted from better WECC RC 
coordination and assistance in real time: 

• A 30-mi nute debate occurred between SCE, which was attempti ng to provide cranki ng power to 
SDG&E to restore SDG&E's system, and the SONGS operators, about the conditions necessary for 
resetting the SONGS separation scheme lockout relay. 

• Recordings showed a lack of clarity among WECC RC, CAISO, and SDG&E about responsibilities 
for restoration efforts. Among other things, this resulted in a SONGS operator making a unilateral 
decision to open a circuit breaker on the line responsible for restoring power to SDG&E's system, 
leaving the line in a less reliable configuration (connected to a single bus). 
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Causes, Findings, And Recommendations 



A. Planning 
Next-Day Planning 



■ Background 

TOPs are required to perform next-day studies to identify and plan for potential 
limitations on their system in the day-ahead timeframe, and to coordinate these studies 
with their neighboring TOPs. 61 These studies provide a proactive mechanism to ensure 
that the system can be operated rel i ably and al low ti me to develop effective operati ng 
solutions. 62 These solutions include, among other things, effective control actions 
needed to return the system to a secure state in anticipated normal and contingency 
system conditions. The development of these plans in the day-ahead timeframe is 
critical because it would be nearly impossible, due to the complexity of the BPS, for 
control room operators to return the system to a secure operati ng state under stressed 
conditions without effective action plans developed in advance. The adequacy of next- 
day studies depends on how extensively and accurately faci I ities and next-day system 
conditions are incorporated into the models used for the studies. This includes 
consideration of a reasonably accurate, current, and complete list of external 
contingencies that could impact a TOP'S system as well asinternal contingencies that 
could impact external SOLs. Consistency of study inputs among all TOPs and BAs is also 
critical for reliable operati on. 

The inquiry found that the affected TOPs' and BAs' procedures for conducting 
next-day studies and models used in these studies vary considerably. As explained more 
fully below, APS does not conduct next-day studies, relying, instead, on two sets of 
studies, conducted on a seasonal and annual basis, that consider a list of possible, 
predetermined contingency scenarios and provide plans to mitigate the contingencies if 
violated. Meanwhile, IID has a policy of conducting next-day studies each day, but 
between April and October of 2011 it failed to perform the required studies on a daily 
basis. Al I other affected TOPs conduct next-day studies, but they use models that do not 



OA See NERC Reliability Standard T0P-002-2b R11. 

See, e.g., NERC Reliability Standard T0P-002-2b ("Current operations plans and procedures are essential to be 
prepared for reliable operations, including response for unplanned events."). 
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adequately reflect next-day operations of faci I ities i n networks external to them. These 
TOPS' next-day studies also do not consider a full list of internal and external 
contingencies that could impose limitations on their daily operations or external 
operations. M oreover, most of these TOPs' next-day studies do not consider the impact 
of sub- 100 kV faci I iti es on BPS rel i abi I ity, such as the i mpact of 1 1 D's CV transformers. 

WECC RC is the highest level of authority responsible for reliable operation of the 
BPS in the Western I interconnection, with the authority to prevent or mitigate emergency 
operating conditions in the next-day and real -time timeframes. As such, WECC RC also 
conducts next-day studies for the entire Western I nterconnection and builds its model 
from the previous day's peak State Estimator case, which includes all facilities operated 
at 100 kV and above and some sub- 100 kV facilities. WECC RC then incorporates 
forecast information, which typically includes transmission outages as provided by 
TOPs, generation outages or derates of 50 M W or greater as provided by TOPs, as well as 
load forecasts, expected net interchange, and unit commitment forecast data from BAs. 
While WECC RC has a more extensive representation of faci I ities throughout theWECC 
footprint in its model than any individual TOP, it does not necessarily monitor or alarm 
for certain lower voltage faci I ities and facilities deemed non-BES that can impact BPS 
rel i abi I i ty. M oreover, because some of the forecasted i nformati on can change between 
the time the TOPs and BAs provide it to WECC RC and the time WECC RC runs its next- 
day studies, WECC RC's next-day studies might not accurately reflect next-day 
operations. 

The September 8th event exposed four weaknesses with the foregoing procedures 
for conducting next-day studies in WECC's region. These weaknesses are detailed in the 
fol lowi ng four fi ndi ngs. A common theme prevai Is i n al I four f i ndi ngs: the affected 
entities do not accurately account for external next-day operating conditions or potential 
external conti ngencies that could i mpact thei r systems. 

Finding 1 Failure to Conduct and Share Next-Day Studies: 

• Not all of the affected TOPs conduct next-day studies or share them with 
their neighbors and WECC RC. Asa result of f ailing to exchange studies, on 
September 8, 20 11 TOPs were not alerted to contingencies on neighboring 
systems that could impact their internal system and the need to plan for such 
contingencies. 

Recommendation 1 

• All TOPs should conduct next-day studies and share the results with 
neighboring TOPs and the RC (before the next day) to ensure that all 
contingencies that could impact the BPS are studied. 
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Failure to Conduct Next-Day Studies 

APS does not conduct next-day studies. I nstead, it relies on two sets of studies, 
conducted on a seasonal and annual basis, for its daily operations. First, APS uses its 
summer and winter seasonal studies for the non-WECC Rated Paths within its 
transmission system. APS performs these studies on a model that it builds from the 
WECC heavy summer base case. I n a coordinated effort with other entities in Arizona, it 
updates this WE CC base case with anticipated loads and resources from the state. APS 
then addsadetailed representation of the entire state's network, including its own 
subtransmission system down to the 12 kV distribution system, to finalize the summer 
model. To create its winter model, APS modifies the summer model with winter peak 
conditions throughout Arizona. 

Once these summer and wi nter models are complete, APS studies a set of 
predetermined contingencies, and relies on the results to determine the response of its 
transmission system to single and multiple contingencies during peak load conditions 
with planned outages modeled. The studies' list of contingencies is based on past 
studies, operating experience, and engineering judgment. The studies also establish 
mitigating measures for contingencies that do not meet loading or voltage guidelines. 

Second, APS relies on a single manual, developed annually, as a guide for its daily 
operations on four Rated Paths within its system. This manual is the result of studies of 
possible, predetermined contingencies on Rated Paths. The results and operational 
instructions in this manual are based on seasonal models that APS develops in 
coordination with four WECC regional study groups, led by CAI SO. CAI SO first sends a 
base case to each study group to update with topology changes for the upcomi ng season. 
Individual members of each study group also update the model with details from their 
systems. CAI SO then i ncorporates al I of the updates and stresses key Paths i n Cal ifornia 
before sendi ng the model back to the study groups. APS uses this model as a starti ng 
point to study the four Rated Paths in its system. APS analyzes the resulting peak-load 
model usi ng a predetermi ned set of si ngle and double conti ngency events that are 
focused primarily on high-voltage transmission outages to determine required actions to 
secure the system for the next most critical N-levent. 63 The manual directs APS to 
rerate relevant Path(s) and identifies necessary mitigating measures as long as the 
contingency (or multiple conti ngency) scenario is included in the manual. The manual, 
however, may not include a particular contingency (or multiple conti ngency) scenario, or 



APS's manual covers only 500 kV and 345 kV facilities, and nothing lower. 
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may not accurately reflect the i nternal and external system topology for the day i n 
question, resulting in the potential for unforeseen circumstances. 

Thus, APS uses seasonal studies for non-Rated Paths and the manual for Rated 
Paths as tools in the day-ahead timeframe, without any additional analysis to validate 
that the tools remain valid for the next day's specific configuration and operation, such 
as transmission or generation outages external to APS's footprint that were not 
anticipated at the time the base seasonal study was performed. APS maintains that these 
tools are sufficient for day-ahead purposes because they include the most severe 
contingencies identified in its system. This viewpoint overlooks the purpose of next-day 
studies— to plan for next-day operations in light of conditions that change daily. By 
relying on tools based on studies conducted on a seasonal and annual basis, APS cannot 
account for all plausible daily scenarios. For typical days that fall within the boundaries 
of the underlying studies and analysis, APS's tools may be viable. For atypical days 
where conditions fall outside the studied boundaries, however, this approach may not be 
adequate. For example, September 8, 20 11 was an atypical day not contemplated by 
APS's manual, as the manual did not account for various generation outages in effect for 
maintenance. 

Between April and October 2011 1 1 D also did not consistently perform adequate 
next-day analyses for each day. Although 1 1 D had a pol icy of conducti ng separate next- 
day analyses for each new day, it fai led to consistently perform the requi red analyses. 
Specifically, IID produced a document each new day showing various changes in 
weather, load and generation forecasts, planned facility outages, potential contingency 
violations, or mitigation measures for identified contingencies, but did not always 
perform the underlyi ng power flow studies for each day between Apri I and October 2011 
On average, between April 20 11 and October 20111 1 D actually performed a study no 
more than two ti mes per week. For the other days, 1 1 D si mply referenced past studies. 
For example, it appears that 1 1 D did not perform a separate, updated study for 
September 8, 2011 because the powerflow study case provided for this day does not 
match the conti ngency results i nd uded i n the dai ly operations guide for the day. I n 
other words, it appears that for September 8, 2011 1 1 D si mply changed the forecasted 
data without actual ly performi ng the next-day study. I nstead, 1 1 D referenced a previous 
study. The referenced study, however, was not valid because it did not match the load 
and generation dispatch data for the day, and there were differences in projected 
overloads reported as potential contingencies. 1 1 D's next-day studies were purportedly 
reviewed by 1 1 D for accuracy, but these discrepancies were not identified. 1 1 D discovered 
thi s i ssue duri ng the course of the i nqui ry and i s i n the process of i mpl ementi ng 
corrective actions to ensure accurate next-day analyses are completed in the future. 
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Finally, the inquiry heard on more than one occasion from TOPs, including APS, 
that WECC RC was responsible for conducting next-day studies or that WECC RC should 
conduct next-day studies that TOPs are currently responsi ble for conducti ng. WECC 
RC's next-day studies for the enti re I nterconnection, however, are not i ntended to 
substitute for the TOPs' next-day studies of their own systems. 

Failure to Effectively Share and Coordinate Next-Day Studies 

I n addition to finding that not all entities conduct next-day studies, the inquiry 
found problems with sharing and coordination among the affected TOPs that do conduct 
such studies. The affected TOPs do not consistently share their studies with neighboring 
TOPs, BAs, and the RC. TOPsgenerally provide studies to WECC RConly if theRC 
identifies an issuein its study and specifically asks to review a TOP'S study. In addition, 
WECC RC's method of sharing its next-day studies with other entities is not effective. 
Specif ical ly, WECC RC's practice is to share the results of its next-day studies when 
conditions warrant or when it receives a request for a study result. 64 WECC RC posts 
on a secure I nternet portal a list of limitations or SOLs identified by its next-day studies 
for i ndividual TOPs and BAs to view, but it is up to TOPs and BAs to access this I ist. 
Also, this list contains only issues that WECC RC deems significant and does not include 
basic, next-day operating conditions, such as scheduled outages. 

One example of the adverse consequences of these shari ng and coord i nation 
issues relates to the 600- pi us MW of TDM generation that was offline for maintenance 
on September 8th. TheTDM generation outage was included in WECC RC's and 
CAI SO's next-day studies, and posted on CAI SO's website, but not incorporated into 
other entities' next-day models and studies. 65 WECC RC receives outage information 
from TOPs and BAs through its Coordinated Outage System (COS). WhileTOPsand BAs 
submit their own information into COS, they cannot access information submitted by 
others. 1 1 D could have benefitted from knowledge of the TDM outages. TheTDM units 
radially connect to the Imperial Valley substation, jointly owned by 1 1 D and SDG&E. If 
theTDM units had been online, they could have mitigated northern I ID overloads on the 



°^ See WECC Reliability Coordination, Operations Planning, Version 3.0, June 22, 2011, at 6, available at 
http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/WECC RC Operations Planning.pdf. 

^ CAISO knew about the outages because the TDM units participate in the CAISO market. CAISO posts daily 
outage unit status reports on its public website that provide the best available data at the date and time of the 
report, for generation units that participate in CAISO's market. These outages are posted at 
http://www.caiso.com/market/Pages/OutageJVlanagement/UnitStatus.aspx . In CAISO's archives, the TDM units 
are shown on outage on September 7 and 8, at a minimum. Dispatch details, however, are not included. WECC 
RC receives CAISO's outage unit status reports daily by email and was aware of the outages. However, IID and 
APS did not know about the TDM outages. 
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CV and Ramon transformers that resulted when H-NG tripped. If 1 1 D had learned about 
these outages from WECC RC or CAI SO, it could have i ncorporated the outages i n the 
day-ahead timeframe and dispatched additional generation, or taken other control 
acti ons, to compensate for the overl oads on its system caused by havi ng these generators 
offl i ne and the H - N G tri ppi ng. 

The September 8th event illustrates that conducting next-day studies and 
sharing the results of such studies are critical to allow TOPs to identify and plan 
for potential contingencies. 

Finding2 Lack of Updated External Networks in Next-Day Study Models: 

• When conducting next-day studies, some affected TOPs use models for 
external networks that are not updated to reflect next-day operating 
conditions external to their systems, such as generation schedules and 
transmission outages. As a result, these TOPs' next-day studies do not 
adequately predict the impact of external contingencies on their systems or 
internal contingencies on external systems. 

Recommendation 2: 

• TOPs and BAs should ensure that their next-day studies are updated to 
reflect next-day operating conditions external to their systems, such as 
generation and transmission outages and scheduled interchanges, which can 
significantly impact the operation of their systems. TOPs and BAs should 
take the necessary steps, such as executi ng nondisclosure agreements, to 
allow the free exchange of next-day operations data between operating 
entities. Also, RCs should review the procedures in the region for 
coordinating next-day studies, ensure adequate data exchange among BAs 
and TOPs, and facilitate the next-day studies of BAs and TOPs. 

As a starting point for their next-day studies, the affected TOPs use models from 
either a TOP'S seasonal basecaseor the previous day's EMS model, if available. The 
seasonal base case represents next-day operati ng conditions i nternal to theTOPs' 
systems, but leaves external networks exactly as they were represented i n the WECC 
seasonal basecase. The affected TOPs' EMS models sometimes includeonly oneor two 
buses outside each TOP'S i nternal footpri nt. Thus, neither type of day-ahead model 
contains actual day-ahead forecasts of system conditions external to each TOP'S system. 
For example, leading into September 8th, the affected TOPs had limited knowledge of 
the current status of transmission facilities, expected generation output, and load 
predictions outside their footprints. Consequently, their next-day studies could not 
adequately predict the impact of external contingencies on their systems or of internal 
contingencies on external systems. 
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1 1 D's next-day study for September 8th i 1 1 ustrates the adverse effects of not 
accounting for external next-day planned operations. 1 1 D used the WECC heavy summer 
seasonal base case to model external conditions for its next-day study for September 8th. 
This base case reflects that most external generation isonlineto meet summer peak 
loads. A heavy summer base case does not accurately represent a shoulder season day 
I i ke September 8th. By September, both generation and transmission mai ntenance had 
started. 

For example, on September 8th TDM generator units in Mexico, totaling more 
than 600 MW, were offline for maintenance. These units are external to 1 1 D and radially 
connect to 1 1 D's jointly owned Imperial Valley substation. When online, this generation 
can help to mitigate overloads on theCV and Ramon transformers in 1 1 D's system. 
Because 1 1 D relied on a heavy summer seasonal model for external networks and did not 
incorporate any updates about the TDM generation, its next-day study did not reflect the 
mai ntenance outage of these units. WiththeTDM generation incorrectly represented as 
being online, 1 1 D's next-day study did not correctly identify how much the loss of H-NG 
would overload 1 1 D's transformers i n its 92 kV system. I n fact, 1 1 D's next-day study for 
September 8, 2011 did not show that the loss of H -NG would overload the CV 
transformers to their trip point. 66 If I ID had learned about theTDM outages (whether 
from CAI SO's website or BY some other method) and i ncorporated the i nformation i nto 
its model, it could have dispatched additional generation, adjusted load, or taken other 
control actions before the loss of H-NG to mitigate such overloading. 

As mentioned above, WECC RC receives next-day data from the entities through 
i nterfaces such as the COS. WECC RC is wel l-situated to faci I itate data-shari ng among 
the 37 BAs and 53 TOPs in the WECC footprint. Given the large number of BAs and 
TOPs in the WECC region, some of which are relatively small in size and resources, 
central coordination and facilitation may be necessary to ensure that all BAs and TOPs 
accurately reflect next-day operating conditions external to their system. 67 WECC RC 
has been worki ng to faci I itate data shari ng by drafti ng and ci rculati ng a universal 



°° The heavy summer base case has more than 1,000 MW more generation in the affected area than was 
available on September 8, 2011. In addition to not representing the offline generation, IID's study, by relying on 
the heavy summer base case, did not accurately reflect the flow on H-NG. The heavy summer base case shows 
flow on H-NG as 1,118 MW, while actual flow on H-NG at the time of the trip was 1,391 MW. 

6 ^ Under current WECC RC procedures, the RC only shares the results of its operational planning analyses if the 
results indicate the need for specific operational actions to prevent or mitigate an instance of exceeding an 
operating limit. WECC Reliability Coordination, Operations Planning, Version 3.0, June 22, 2011, at 6, available at 
http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/WECC RC Operations Planning.pdf. 
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nondisclosure agreement. As this report was being finalized, less than 30 of the 
approximately 100 discrete entities within WECC had signed the agreement. 68 

Finding3 Sub- 100 kV Facilities Not Adequately Considered in 
Next-Day Studies: 

• I n conducting next-day studies, some affected TOPs focus primarily on the 
TOPS' internal SOLs and the need to stay within established Rated Path 
limits, without adequate consideration of some lower voltage facilities. Asa 
result, these TOPs risk overlooking facilities that may become overloaded 
and impact the reliability of the BPS. Similarly, the RC does not study sub- 
100 kV facilities that impact BPS reliability unless it has specifically been 
alerted to issues with such facilities by individual TOPs or the RC has 
otherwise identified a particular sub- 100 kV facility as affecting the BPS. 

Recommendation 3: 

• TOPs and RCs should ensure that their next-day studies include all internal 
and external facilities (including those below 100 kV) that can impact BPS 
reliability. 

The September 8th event showed that some sub- 100 kV faci I ities can have 
significant i mparts on BPS rel iabi I ity, such as causi ng i nstabi I ity or cascadi ng outages. 
Yet, it appears that these faci I ities are not adequately considered i n the day- ahead 
timeframe. For example, 1 1 D's 92 kV network runs parallel to two major transmission 
Paths: (1) Path 44, which connects to the SW PL via the Palo Verde- Devers 500 kV line 
(part of Path 49) and runs to the north of 1 1 D; and (2) the SWPL, which runs to the south 
of 1 1 D. Given the paral lei nature of its system, 1 1 D's 92 kV system is forced to carry a 
significant portion of any east- west power flows whenever segments of Path 44 or the 
SWPL are out of service. 

Because none of the affected TOPs, besides 1 1 D, considered 1 1 D's 92 kV network 
in their next-day studies, they were not aware how their internal contingencies could 
affect 1 1 D's 92 kV network, or how an overload on 1 1 D's 92 kV network could affect their 
systems. For example, APS does not routi nely study 1 1 D's lower voltage faci I ities, 
i ncl udi ng the CV and Ramon transformers, i n the day-ahead ti meframe. APS uses 
seasonal studies and its operations manual as its tools in the day-ahead timeframe. 
Whilethe model used for the seasonal studies physically has 1 1 D's 92 kV network 
represented, neither the model nor the operations manual are used to consider the next 
day's specific configuration and operation, such as transmission or generation outages 
external to APS's footpri nt that were not anti ci pated at the ti me the seasonal study and 



The agreement does address market concerns by requiring entities who participate in data-sharing to respect 
the separation of market and operations functions. 
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manual were updated. As a result APS was not ableto predict what occurred on 1 1 D's 
system— increased flows and overloading on its 92 and 161 kV transformers and 
transmission lines— when H-NG tripped offline. Similarly, affected TOPs other than IID 
do not consider in their day- ahead planning how the loss of the CV and Ramon 
transformers, leading to the S Line RAS operation, could adversely affect their internal 
systems. Accordingly, TOPs should revise their next-day study practices to account for 
all facilities, including those operated below 100 kV, that impact BPS reliability. 

WECC RC also did not adequately consider sub- 100 kV facilities not identified as 
BES that can have significant impacts on BPS reliability. Whi le WECC RC does model 
1 1 D's CV transformers in its next-day studies, prior to September 8, 201X it did not "flag" 
them in its studies for active monitoring. 69 This means that WECC RC had data showing 
that the transformers would overload under certain conditions, but the overloads were 
not identified by alarms to be seen by RC operators. WECC RC did not actively monitor 
the CV transformers in its next-day studies because they are below 100 kV and 1 1 D had 
not alerted WECC RC to any issues that would warrant monitoring of the transformers. 
Given theCV transformers' impact on BPS reliability, WECC RC should actively monitor 
these transformers. 70 

Finding 4 Flawed Process for Estimating Scheduled Interchanges: 

• WECC RC's process for estimating scheduled interchanges is not adequate to 
ensure that such values are accurately reflected in its next-day studies. Asa 
result, its next-day studies may not accurately predict actual power flows and 
contingency overloads. 

Recommendation 4: 

• WECC RC should improve its process for predicting interchanges in the day- 
ahead timeframe. 

I nterchanges are energy transfers that cross BA Areas. I nterchanges can affect 
flows across transmission systems, so forecasting accurate interchanges is important in 
the day-ahead timeframe to plan for potential overloading. WECC RC's process for 
estimating scheduled interchanges is not adequate to ensure that the scheduled 
i nterchanges i ncorporated i nto its next-day studi es are accurate. U nder thi s process, by 
10:00 AM each day BAs provide WECC RC with all interchanges they have approved for 



M To aid in effectively and efficiently processing and analyzing reliability data for the entire Western 
Interconnection, WECC RC has the option of flagging a subset of facilities for active monitoring in its studies. It 
has since updated this feature to flag the CV transformers for monitoring. 

^ WECC RC has implemented new procedures since September 8, 2011, to monitor RTCA results for the CV 
transformers. 
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the next day. The B As typically submit this information once per day without any 
subsequent updates. WECC RC then validates these scheduled interchanges by 
comparing the values with what the BAs provided the prior day and with what WECC 
RCs state esti mator observed i n the prior days and weeks. 

The accuracy of interchange data in WECC RCs next-day studies could be 
i mproved by al lowi ng for updates closer to real ti me. BAs' i nterchange data are I i kely to 
change after their 10:00 AM submittal to WECC RC. Some BAs have automated 
systems, which send updates of i nterchange data to WECC RC. M ost BAs submit the 
data manually, only once at 10:00 AM . I nclusion of a process or requirement for BAs to 
update their scheduled interchanges after their 10:00 AM submission would increase the 
I i kel i hood of accurate i nterchange data. 

The accuracy of i nterchange data affected WECC RCs next-day study for 
September 8, 2011 Specifically, the scheduled interchanges reflected in WECC RCs 
next-day study for September 8, 201X were not sufficiently accurate to predict that 1 1 D's 
CV 230/92 kV transformers would overload to their trip point upon the loss of H-NG. 
After the event, WECC RC ran its next-day study using actual interchanges, and found 
that the CV transformers would overload beyond their tripping threshold upon the loss 
of H -NG. If WECC RC had used more accurate net i nterchange data and flagged the CV 
transformers for monitoring, it could have learned of the issues with these transformers 
and alerted 1 1 D or issued directives for control actions to mitigate the situation, such as 
increasing generation or shedding load. 

Seasonal Planning 



■ Background 

Following a set of disturbances in the Western I nterconnection during the 
summer of 1996, WECC established a new seasonal planning structure designed to avert 
system- wide disturbances whi le maxi mizi ng the commercial avai labi I ity of transmission 
capacity. This new structure involved the creation of the Operating Transfer Capability 
Policy Committee (OTCPC). The purpose of theOTCPC was to provide coordinated 
standard development and determination of seasonal Operating Transfer Capabilities 
(OTCs), or Operating Transfer Limits, 71 within the Western I nterconnection. 72 



OTCs are now known as SOLs. 

72 The OTCPC itself was abolished and replaced with a new structure in June 2011; however, planning for the 
seasonal period in which the blackout occurred was performed under the OTCPC structure, so the inquiry's 
analysis focused on the OTCPC structure. 



-72- 



FERC/NERC Staff Report on the September 8, 2011 Blackout 



Among other things, the OTCPC was designed to be responsible for determining 
which transmission Paths should be studied, facilitating OTC dispute resolution, 
ensuring that seasonal studies maintain consistent standards and methodologies, and 
approvi ng seasonal studies of OTC I i mits. To that end, the OTCPC was charged with 
reviewing and approving study plans and technical simulation results; developing 
policies and procedures addressing seasonal OTCs; establishing working groups such as 
subregional study groups and the Operating Procedures Review Group; addressing OTC 
seams issues between subregions; and providing technical guidance. 

The seasonal study plans that are reviewed and approved by the OTCPC were 
created by a set of four subregional study groups (someti mes referred to as SRSGs or 
simply subregions). There were four groups: (1) the California/ Mexico Operations 
Study Subcommittee (OSS); (2) the Northwest Operational Planning Study Group 
(NOPSG); (3) the Rocky Mountain Subregional Study Group (RMSG); and (4) the 
Southwest Area Study Group (SASG). The affected entities were members of two of 
these groups: the OSS (CAI SO, SDG&E, SCE, CFE, and 1 1 D) and the SASG (APS, 
WALC). 

On an annual basis, each subregional study group reviewed the Paths in its 
subregion to determine which Paths should be studied and the system conditions under 
which they should be studied. Then, seasonally, the four subregional study group chairs 
submitted their recommendations of which Paths to study to the OTCPC for review and 
approval. Following OTCPC's approval, the studies were performed in accordance with 
the OTC study process. This process began with establishment of an initial "base case" 
by WECC staff, with input from representatives of each subregional group. The "base 
case" is a computer model of projected or starting power system conditions for a specific 
point in time. For the 2010-2011 planning year, five base cases were used. 73 Once the 
comments from thefour subregional representatives were incorporated, the final cases 
were made avai I abl e vi a W E CCs web si te for adj ustment and mod i f i cati on by 
subregional members in order to forecast expected seasonal conditions on the system. 
The subregional members performed their own seasonal studies, and then met to discuss 
the results. A subregional seasonal planning case was produced on this basis, but no 
further studies were performed. Subregional seasonal cases were shared among thefour 
subregions via liaisons from the other subregions. No comprehensive WECC-wide Path 
rating study was prepared on the basis of thefour subregional studies. 



These included low summer load, high summer load, low winter load, high winter load, and high spring load. 
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In addition to, and apart from, the seasonal planning studies just described, 
TOPs also conduct their own seasonal studies focusing on their own internal networks. 
Theseinternal studies follow a different process from the seasonal Path rating studies, 
though both begin with the WECC base case. I nternal seasonal studies, however, are not 
aggregated or reviewed at thesubregional level. I nstead, TOPs generally replace the 
i nformation from the WECC base case with more accurate and granular detai I for thei r 
own areas only. Once updated, the TOPs perform contingency analyses for their own 
internal purposes. Theythen share with their neighbors the results of these operational 
studies, which typically contain only the default data from the WECC base case for 
everythi ng outsi de of thei r own areas. 

The inquiry identified a number of issues relating to both types of seasonal 
planning bythe affected entities. These issues impaired the accuracy and effectiveness of 
the seasonal studies by excluding, in various ways, pertinent issues and information that 
should have been taken into consideration. 

Finding5 Lack of Coordination in Seasonal Planning Process: 

• The seasonal planning process in the WECC region lacks effective 
coordination. Specifically, the four WECC subregionsdo not adequately 
integrate and coordinate studies across the subregions, and no single entity 
is responsible for ensuring a thorough seasonal planning process. Instead of 
conducting a full contingency analysis based on all of the subregions' studies, 
the subregions rely on experience and engineering judgment in choosing 
which contingencies to discuss. As a result, individual TOPs may not identify 
contingencies in one subregion that may affect TOPs in the same or another 
subregion. 

Recommendation 5: 

• WECC RE should ensure better integration and coordination of the various 
subregions' seasonal studies for the entire WECC system. To ensure a 
thorough seasonal planning process, at a minimum, WECC RE should 
require a full contingency analysis of the entire WECC system, using one 
integrated seasonal study, and should identify and eliminate gaps between 
subregional studies. I ndividual TOPs should also conduct a full contingency 
analysis to identify contingencies outside their own systems that can impact 
the reliability of the BPS within their system and should share their seasonal 
studies with TOPs shown to affect or be affected by their contingencies. 

No comprehensive WECC-wideseasonal studies are performed. With respectto 
seasonal Path rating studies, a representative or leader from each subregion adapts the 
WECC base case on the basis of input from subregional members, and then makes these 
revised cases avail able to the other subregional members for review, comment, and 
approval. Thesubregional leader then conducts the seasonal studies concentrating only 
on the rated Paths in the subregion. The results of the seasonal Path rating studies are 
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shared and discussed first among the subregion's members, and then with the other 
subregions, but neither WECC RE nor theOTCPC performs or mandates any further 
seasonal studies, and no new WECC-wide seasonal study is performed to reflect the 
input of all of thesubregions. Instead, representatives of the subregional groups gather 
informallyto discuss the results of their seasonal studies and rely on experience and 
engineering judgment to identify and resolve any issues. 

The events of September 8, 2011 illustrate that this process is not adequate: the 
tripping of one line in a rated Path— H-NG, which is part of Path 49— ultimately led to 
the tripping of other lines in other rated Paths, including Paths 44 and 45. Focusing 
exclusively on Path ratings— and solely on a subregional basis— ignores network facilities 
that can impact rated Paths (and vice- versa) and does not account for the 
interrelationships of Paths and other facilities across WECC's subregions. 

With respect to the internal seasonal studies, there is even less coordination. 
TOPs generally perform internal seasonal studies using models that include detailed data 
for thei r own system, but default to WECC base case data, which may not be sufficiently 
detai led or updated, for everythi ng else. TOPs perform conti ngency analysis for thei r 
own internal areas using this model. Nostudy is done to identify the impact of external 
contingencies on the TOP'S system, or the impact of the TOP'S internal contingencies on 
the SOLs of other TOPs. TOPs provi de the results of their internal seasonal studiesto 
neighboring TOPs for informational purposes, after which those TOPs mayor may not 
provide comments. 

In all, this situation indicates that theTOPs' internal seasonal planning studies 
are too heavily reliant upon the assumptions underlying and reflected in a single WECC 
base case, and do not consider and study i mparts of variations from that base case. 

The September 8th event demonstrated one example where better i ntegration of 
seasonal studies across two subregions is needed. When H-NG (part of Path 49) tripped, 
approximately 12% of the flow from that line, which is located in theSASG subregion, 
was transferred across 1 1 D's 230/ 92kV transformers, via the 1 1 D 92kV local network to 
the southern IID 161 kV network, which are all in the OSS subregion. This additional 
flow on 1 1 D's CV transformers ultimately resulted in cascading outages and imparted 
Paths 44 and 45. The affected entities were unaware of this potential inter-Path impart, 
because the SASG and OSS studies had not been jointly considered. Moreover, since the 
subregional studies concentrate only on Path ratings, this flow transfer was not 
apparent. I f the seasonal studies of SASG and OSS had been better coordi nated and 
more rigorously analyzed, the potential for the loss of H-NG to overload 1 1 D's 92 kV 
network could have been identified and mitigation plans developed. 
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Finding 6 External and Lower-Voltage Facilities Not Adequately Considered 
in Seasonal Planning Process: 

• Seasonal planning studies do not adequately consider all facilities that may 
affect BPS reliability, including external facilities and lower-voltage 
facilities. 

Recommendation 6: 

• TOPs should expand the focus of their seasonal planning to include external 
facilities and internal and external sub- 100 kV facilities that impact BPS 
reliability. 

As noted above, TOPs performing subregional Path rating studies do not 
sufficiently account for the i mpact of faci I i ties external to thei r subregi on, or faci I i ties 
within their subregi on that are not part of a rated Path. Moreover, no WECC-wide Path 
rating study is performed to harmonize and analyze the impact of one subregi on on the 
rest of the subregi ons. 

The problem with this approach is illustrated in the example cited above: The 
tripping of a part of one rated Path, H-NG, which is part of Path 49, led to the tripping of 
portions of other rated Paths. The mechanism whereby these other trips were triggered 
was the transfer of flow across low- voltage (below 100 kV) facilities that were located in a 
different subregi on. Under the approach to Path rating studies in pi ace at the time, it 
would have been impossiblefor WECC RE or TOPs to anticipate and study this 
possibility, because it occurred across subregi ons, indirectly, via I ower-voltage facilities. 
Even if seasonal Path rating studies had been performed across subregions, these studies 
would not have anticipated this possibility, unless they also took into account lower- 
voltage facilities, which they presently do not. 

The i nternal seasonal pi anni ng studi es of the vari ous TOPs are subj ect to si mi I ar 
omissions, although these studies encompass more than just the rated Paths and contain 
more detail than the Path rating studies. The practices of individual TOPs differ, but 
none contains sufficient detail and accuracy with respect to facilities outside their own 
footprints, as well as I ower-voltage facilities. 1 1 D, for example, has explained that it 
"does not identify or study components outside of the 1 1 D territory below 100 kV for 
impacts on the BPS reliability in its territory," nor does it "identify or study components 
i nside of the 1 1 D territory below 100 kV for i mparts on the BPS rel iabi I ity outside of its 
territory." 

Similarly, while CAI SO studies in its seasonal planning process "all of the 
transmission components that it operates, some of which are below 100 kV," it has also 
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acknowledged that it "does not have the necessary information to accurately study 
transmission components below 100 kV outside of its territory to determi ne if they have 
an impact on the BPS reliability in [CAI SO's] service territory." 

The events of September 8, 20 IX demonstrate that sub- 100 kV f aci I i ti es i n 
parallel with BPS systems can have a significant effect on BPS reliability. The loss of H- 
NG caused the overloading and tripping of both 230/92 kV transformers at CV, which in 
turn caused another sub- 100 kV transformer to trip at Ramon, which led to the 
cascadi ng outages di scussed i n detai I above. This possi bi I i ty was not studi ed as part of 
the seasonal studies by any of the TOPs, other than 1 1 D, because the CV transformers' 
secondary windings are below 100 kV. The seasonal studies conducted by affected TOPs, 
other than 1 1 D, did not study the i mpact of the CV transformers. I f the CV transformer 
contingency overloads had been identified as limiting elements in the seasonal plans, the 
cascading outages might have been avoided or lessened by having pre-contingency 
mitigation in place, such as increasing generation on 1 1 D's 92 kV system. 

Finding 7 Failure to Study Multiple Load Levels: 

• TOPs do not always run their individual seasonal planning studies based on 
the multiple WECC base cases (heavy and light load summer, heavy and light 
load winter, and heavy spring), but, instead, may focus on only one load 
level. Asa result, contingencies that occur during the shoulder seasons (or 
other load levels not studied) might be missed. 

Recommendation 7: 

• TOPs should expand the cases on which they run their individual planning 
studies to include multiple base cases, as well as generation maintenance 
outages and dispatch scenarios during high load shoulder periods. 

WECC created five base cases for the 2010-2011 season— heavy and light load 
summer, heavy and light load winter, and heavy spring— intended to capture the 
spectrum of possible loading configurations at different times of the year. The inquiry 
found that some of the affected TOPs deemed it unnecessary to run individual planning 
studies based on the multiple WECC base cases. I nstead, these TOPs identified some 
subset of these base cases that they concl uded were most relevant to thei r concerns and 
ran studies based on only that subset of base cases. Some TOPs employed only one base 
case— the heavy load summer base case— for planning the season during which the 
September 8, 2011 blackout occurred. By limiting the run of planning studies to a small 
subset of base cases, TOPs restrict thei r abi I ity to antici pate and respond to 
contingencies arising in the context of load levels that vary significantly from those in the 
subset of base cases upon which thei r studies were predicated. 
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As noted above, September 8, 2011 was a very hot day in the region, and 
scheduled flows in the 1 1 D footprint were near record peaks. The high demand on 
September 8th was indeed similar to what would have been modeled in a heavy load 
summer seasonal study. The generation picture, however, was very different. By 
September 8, 20 11 generation maintenance— which is not typically scheduled for 
summer peak days— had begun. The "heavy peak" summer study base cases that were 
actually used for September 8 th therefore had built into them the incorrect assumption 
that there would be minimal maintenance— i.e., that most generation would be on line — 
and thus did not account for the normal resumption of facility maintenance in the 
shoulder season. 

If 1 1 D's seasonal studies had assumed even a modest decrease in the available 
generation, they might have enabled 1 1 D to antici pate and prevent the events that 
occurred on its system. 1 1 D was unaware of theTDM mai ntenance outages, but if it had 
conducted a shoulder season study, it might have been operating in a mode that more 
accurately reflected actual operating conditions on that day and could have potentially 
avoided the overloadi ng of CV transformers to the tri ppi ng poi nt. This lack of awareness 
i 1 1 ustrates the ri sks of not separately model i ng the shoul der months such as September, 
when facility maintenance has begun but demand could remain or become very high. 
During these times, generation to serve load may come from other areas, changing flow 
patterns from those that typically occur on a normal summer peak day in which most 
generation is on line. 

Finding8 Not Sharing Overload Relay Trip Settings: 

• I n the seasonal planning process, at least one TOP did not share with 
neighboring TOPs overload relay trip settings on transformers and 
transmission lines that impacted external BPS systems. 

Recommendation 8: 

• TOPs should includein the information they share during the seasonal 
planning process the overload relay trip settings on transformers and 
transmission lines that impact the BPS, and separately identify those that 
have overload trip settings below 150% of their normal rating, or below 115% 
of the highest emergency rating, whichever of these two values is greater. 

As discussed in greater detail below, the relay trip settings of 1 1 D's CV 230/92 kV 
transformers were set very low, j ust above the faci I i ties' emergency rati ng. These 
setti ngs effectively meant that 1 1 D's system operators had very I ittle ti me to respond to 
the overload resulting from the loss of H-NG beyond emergency ratings and could not 
rely on post-contingency mitigation. If 1 1 D's neighbors had been aware of the relay trip 
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settings on these transformers when preparing their seasonal studies, they would have 
been able to plan for the possibility of the CV transformers tripping at a lower trip point. 

As a general matter, TOPs shoul d be aware of the rel ay tri p setti ngs of faci I i ti es i n 
neighboring areas that have the potential to impact portions of the BPS within their own 
areas, regardless of whether or not those facilities have been defined as, or deemed to be, 
BES facilities. This concern is particularly acute where the overload trip points of the 
faci I ity i n question are set below 150% of thei r normal rati ng, or below 115% of thei r 
emergency rating, because, as discussed below, such settings sharply limit the amount of 
ti me avai lable for operators to i mplement post-conti ngency mitigation measures. These 
settings require that all entities that could be affected are aware and ableto implement 
pre-conti ngency mitigation. 

Near-and Long-Term Planning 



■ Background 

TPs and PCs conduct near- and long-term studies to ensure their systems are 
planned for reliable operation under normal operating conditions. I n addition, the 
system faci I ities must remai n stable i n the event of si ngle and multi pie conti ngency 
scenarios. Near-term studies consider potential contingencies one to five years past the 
study date, and long-term studies consider potential contingencies six to ten years past 
the study date. The near- and long-term planning process in theWECC region involvesa 
coordinated effort among individual TPs and PCsatthelocal level, Subregional Planning 
Groups (SPGs) 74 at the regional level, and WECC RE at the Interconnection- wide level. 
It is a multi-step process, performed annually. 

First, TPs and PCs submit data about their internal networks to their respective 
SPG for each horizon year studied (i.e., years onethrough ten). These data include 
forecasted load levels and facilities projected to be in or out of service. Also, these data 
assume peak load conditions and, thus, reflects that most internal generation is online. 
Second, SPGs add information to these data based on their broad knowledge of planning 
projects and reliability issues within their respective regions. For example, an SPG 



m There are five SPGs in the WECC region, each representing a specific area and composed of various members 
and stakeholders, including individual owners and operators of transmission networks, representatives of local 
government agencies, and independent developers. SPGs allow for the joint consideration of issues among 
individual members. APS, IID, and WALC are members of WestConnect, which performs the SPG function in the 
Southwest region. SDG&E and SCE are members of CAISO, which performs the SPG function in parts of California. 
The SPGs are involved in near- and long-term planning only and are unrelated to the SRSGs, discussed above, 
which deal with seasonal planning. 
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might add data for a particular horizon year based on its knowledge of a merchant 
generator's desire to connect to the grid. SPGs also consider future projects needed for 
reliability and the effect of environmental regulations on the future operation of 
generator units. Thi rd, SPGs merge al I of thei r members' cases to create a regional case. 
Fourth, WECC RE merges the various regional cases from all the SPGs to create the base 
case for each horizon year. WECC RE makes these cases availableon its website for TPs, 
PCs, and SPGs to access. Finally, TPs and PCs add their own subtransmission facilities 
to the base cases to run their near- and long-term studies. TPs and PCs typically choose 
a I i st of conti ngenci es to study based on past experi ence and engi neeri ng j udgment. 



As discussed below, this multi-step process has several shortcomings, which left 
the affected entities unprepared for the September 8th event. 



Finding9 Gaps in Near- and Long-Term Planning Process: 

• Gaps exist in WECC RE's, TPs' and PCs' processes for conducting near- and 
long-term planning studies, resulting in a lack of consideration for: (1) 
critical system conditions; (2) the impact of elements operated at less than 
100 kV on BPS reliability; and (3) the interaction of protection systems, 
including RASs. As a consequence, the affected entities did not identify 
during the planning process that the loss of a single 500 kV transmission line 
could potentially cause cascading outages. Planning studies conducted 
between 2006 and 20 11 should have identified the critical conditions that 
existed on September 8th and proposed appropriate mitigation strategies. 

Recommendation 9: 

• WECC RE should take actions to mitigate these and any other identified gaps 
in the procedures for conducting near- and long-term planning studies. The 
September 8th event and other major events should be used to identify 
shortcomings when developing valid cases over the planning horizon and to 
identify flaws in the existing planning structure. WECC RE should then 
propose changes to improve the performance of planning studies on a 
subregional- and I nterconnection-wide basis and ensure a coordinated 
review of TPs' and PCs' studies. TOPs, TPs and PCs should develop study 
cases that cover critical system conditions over the planning horizon; 
consider the benefits and potential adverse effects of all protection systems, 
including RASs, Safety Nets (such as the SONGS separation scheme), and 
overload protection schemes; study the interaction of RASs and Safety Nets; 
and consider the impact of elements operated at less than 100 kV on BPS 
reliability. 

The affected entities' near- and long-term planning studies for horizon year 2011 
(i.e., the studies conducted in 2001through 2010) did not identify that the loss of a 
single500 kV linein APS's territory would cause cascading outages across the territories 
of SDG&E, CFE, 1 1 D, and WALC. Several gaps in the near- and long-term planning 
process contributed to these omissions. First TPs and PCs submit peak load data to 
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WECC for i ncorporati on i nto the base case and, thus, the data assume that most i nternal 
generation isonlineto meet peak conditions. Asa result, themodelsfor 2011did not 
contain accurate, realistic representations of online generation. Running studies under 
the assumption that most generation is online provided an unrealistic portrayal of 
system transfers on the day of the event. 

I ndeed, system transfers fol lowi ng the loss of H - NG were higher than the 
transfers seen i n the base case used for near- and long-term studies. Significant flows 
from H-NG transferred across 1 1 D's and WALC's systems and onto Path 44. Flow on 
Path 44 i ncreased by approxi mately 84% fol lowi ng the loss of the I i ne. These large 
system transfers went undetected in near- and long-term studies, and the affected 
entities were not alerted to the need to plan for these critical system conditions. To avoid 
this problem in thefuture, TPs and PCs should study more generation dispatch scenarios 
to provi de a more real i sti c proj ecti on of system transfers fol I owi ng conti ngenci es. 

Second, TPs and PCs do not run a full list of external contingencies during the 
near- and long-term planning process. I nstead, they rely on experience and engineering 
judgment, focusing on previously identified contingencies. This can be particularly 
problematic in today's operating environment in which the nature and limitations of the 
system are rapidly changing. For example, as part of its near- and long-term planning 
1 1 D studied potential contingencies on four WECC Rated Paths, but did not study the 
loss of H -NG. As a result, 1 1 D was not prepared for the effect on its system when that 
line tripped. Also, while I ID'sCV 230/92 kV transformers are included in the base case, 
some of the affected TPs and PCs did not study the potential loss of these facilities. By 
not consider i ng a complete I ist of external conti ngenci es that could i mpact thei r systems, 
TPs' and PCs' studies for horizon year 2011 were not sufficient to identify and plan for 
the impact of external contingencies on their internal systems or internal contingencies 
on neighboring systems. 

Third, TPs and PCs do not study external subtransmission facilities in the near- 
and long-term planning process. I ndividual TPs and PCs add their own subtransmission 
faci I ities after the base case has been created by WECC RE, but do not add external 
subtransmission equipment. If external subtransmission systems were included in the 
base case, entities could identify the parallel flow on such lower-voltage systems that can 
result from transmission contingency outages. This consideration is particularly 
important for lower voltage systems that parallel external high voltage systems. For 
example, when APS's H-NG tripped, approximately 12% of its flow transferred to 1 1 D's 
92 kV system. This i ncreased flow and overloadi ng on 1 1 D's system had a ri pple effect, 
causi ng cascadi ng outages throughout neighbori ng territories. Because the affected 
entities did not study external subtransmission systems in their near- and long-term 
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studies, they did not identify the potential for overloading on 1 1 D's 92 kV system or the 
impact on their systems from this overloading. 



Fourth, TPs and PCs do not sufficiently study the interaction of protection 
systems in external networks in their near- and long-term planning studies. For 
example, some of the affected TPs and PCs did not study the i nteraction between the 
overload protection on 1 1 D's three 230/ 92 kV transformers, or between the protection on 
these transformers and the S Line RAS. Based on the pre-event conditions, the loss of 
oneCV transformer would automatically result in the loss of the second, followed 
automatically by the loss of the Ramon transformer, which in turn, would result in either 
voltage col I apse and load shedding, or overloading on the S Line. The S Line RAS is 
desi gned to mi ti gate overl oads by tri ppi ng generati on i n M exi co that suppl i es power to 
1 1 D. H owever, operation in this manner only served to further overload 1 1 D and WALC 
faci I ities and exacerbate system conditions on the day of the event. The affected entities 
should have studied the interaction of these schemes to prepare for the impacts on their 
systems. 



Finding ID Benchmarking WECC Dynamic Models: 

• The inquiry obtained a very good correlation between the simulations and 
the actual event until the SONGS separation scheme activated. After 
activation of the scheme, however, neither the tripping of the SONGS units 
nor the system collapse of SDG&E and CFE could be detected using WECC 
dynamic models because some of the elements of the event are not explicitly 
included in those models. Sample simulations of the islanded region showed 
that by adding known details from the actual event, including UFLS 
programs and automatic capacitor switching, the simulation and event 
become more closely aligned following activation of the SONGS separation 
scheme. 

Recommendation ID: 

• WECC dynamic models should be benchmarked by TPs against actual data 
from the September 8th event to improve their conformity to actual system 
performance. In particular, improvements to model performance from 
validation would be helpful in analysis of under and/ or over frequency 
events in the Western I nterconnection and the stability of islanding 
scenarios in the SDG&E and CFE areas. 

The i nqui ry si mulated the dynamic system response of the September 8th event 
from prior to the loss of H-NG through the separation of Path 44 and the unsuccessful 
islanding of SDG&E and CFE. The team obtained very good correlation between the 
simulation model and the actual event until the SONGS separation scheme activated. 
H owever, neither the tri ppi ng of the SONGS units nor the system col lapse of SDG&E and 
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CFE could be predicted using existing WECC dynamic models entities use to perform 
near- and long-term planning. 

This inability to use the existing system models to reproduce the actual event is 
also evident in the post-event analysis that was prepared by SDG&E on the effectiveness 
of UFLS programs following the September 8th event. 75 The SDG&E post-event 
analysis shows that the UFLS performance should have prevented the SDG&E system 
from frequency col I apse, similar to the "as is" results shown in Figure 1A, below. 
However, the SDG&E analysis does not explain why the simulation results are so 
different than the actual system responses— i.e., successful islanding operation versus 
system collapse. 



Figure 14: Actual and Simulated Frequency at Miguel 500 kV Bus 




The inquiry's Modeling and Simulation team was able to obtain a simulation 
more closely aligned with actual measured performance by performing several sensitivity 
studies and adding details from the actual event, includingUFLS performance, PMU 



3 Preliminary Analysis of SDG&E Off-Nominal UFLS Program Effectiveness Following September 8, 2011 Pacific 
Southwest Event, Performed by SDG&E, December 7, 2011. 
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data, and generation tripped in CFE's and SDG&E's territories. For example, one 
sensitivity study (referred to here as 'Test 3") simulated approximately: 



a) 3,080 M W of UFLS in SDG&E 13 seconds after Path 44 tripped (compared to 2,760 M W in 
"as- is" case) 

b) 520 MW of UFLS in CFE after Path 44 tripped, but prior to SDG&E separation from CFE/ APS 
(compared to 900 M W modeled in "as- is" case) 

c) 589 M W of generation tri pped i n CFE after Path 44 tri pped, but prior to SDG&E separation 
from CFE/ APS (compared to zero in "as- is" case) 

d) 1000 MW of generation tripped in SDG&E immediately after SDG&E separated from 
CFE/ APS (compared to zero in "as-is" case) 



Figure 15, below, shows results of 'Test 3." As can be seen, this simulation 
more closely follows the actual event than the "as-is" model used in Figure 14. 



Figure 15: Miguel Frequency Actual and Simulated for 'Test 3' 
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The simulation studies explain the ineffectiveness of the UFLS program, despite 
up to 75% of SDG&E load that was shed withi n 13 seconds of the SONGS separation 
scheme operating. Thesimulation analysi s confirmed findings in the inquiry's SOE that 
the frequency col I apse was caused by generation trips and UFLS mi soperations within 
CFE shortly after Path 44's separation, followed by additional generation trips within 
SDG&E around the time it separated from CFE/APS. 
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B. Situational Awareness 
Background 

TOPs, BAs, and RCs have system operators who constantly monitor thei r 
networks to maintain situational awareness of system conditions, identify potential 
system disturbances, and institute mitigating measures, as necessary. The affected 
entities utilize a range of tools to perform these functions. All of the entities useSCADA 
systems as thei r mai n monitori ng tool . SCADA systems typical ly consist of a central 
computer that receives information from various RTUs and intelligent electronic devices 
(I EDs), located throughout the system. SCADA systems provide control center operators 
with real-ti me measurements of system conditions and can send alarms to signal a 
problem. 

M ost of the affected entities also use several other tools to study and analyze the 
i nformation received from thei r SCADA systems. Two of the most i mportant tools are 
State E sti mator and RTCA. State E sti mator gathers the avai I abl e measurements from 
the SCADA system and calculates estimated real-ti me values for the whole system. 
RTCA then takes the information from State Estimator and studies "what if" scenarios. 
For example, RTCA determines the potential effects of losing a specific facility, such asa 
generator, transmission line, or transformer, on the rest of the system. I n addition to 
studying the effects of various contingencies, RTCA can prioritize contingencies. It can 
also provide mitigating actions and send alarms (visual and/ or audible) to operators to 
alert them to potential contingencies. 

While most of the affected entities have and use these tools, the inquiry identified 
several concerns with entities' ability to adequately monitor, identify, and plan for the 
next most cri ti cal conti ngency i n real ti me. Several areas for i mprovement are descri bed 
in the findings below. 

PM Us did not play a role in observing the September 8th event in real time, but 
may prove i ncreasi ngly i mportant i n situational awareness. Of the affected entities, 
CAI SO, SCE, and APS are equipped with PM Us. PM Us are widely distributed 
throughout WECC as the result of a WECC-wide initiative known as the Western 
I nterconnection Synchrophasor Program (Wl SP). Their high sampling speed (up to 30 
samples per second) and excellent GPS-based time synchronization offer new granularity 
in information about voltage phase angles and other grid conditions. PM Us are expected 
to be used to identify and monitor for grid stress, grid robustness, dangerous 
oscillations, frequency instability, voltage instability, and reliability margins. While not 
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yet sufficiently integrated to have been used by the affected entities in their control 
rooms on September 8th, as discussed earlier, PM U data proved valuable in constructing 
the sequence of events and other post-event analysis. 

Finding 11 Lack of Real-Time External Visibility: 

• Affected TOPs have limited real-time visibility outside their systems, 
typically monitoring only one external bus. As a result, they lack adequate 
situational awareness of external contingencies that could impact their 
systems. They also may not fully understand how internal contingencies 
could affect SOLs in their neighbors' systems. 

Recommendation LL 

• TOPs should engage in more real-time data sharing to increase their 
visibility and situational awareness of external contingencies that could 
impact the reliability of their systems. They should obtain sufficient data to 
monitor significant external facilities in real time, especially those that are 
known to have a direct bearing on the reliability of their system, and 
properly assess the impact of internal contingencies on the SOLs of other 
TOPs. In addition, TOPs should review their real-time monitoring tools, 
such as State Estimator and RTCA, to ensure that such tools represent 
critical facilities needed for the reliable operation of the BPS. 

Although al I of the affected TOPs use SCADA to monitor thei r own systems, some 
TOPs' situational awareness is hindered by their limited visibility into neighboring 
systems. Some of the affected TOPs' real-ti me external visi bi I ity is I i mited to one or two 
buses outside their systems. The September 8, 2011 event demonstrated that more 
expansi ve vi si bi I i ty i nto nei ghbori ng systems i s necessary for these TOPs to mai ntai n 
situational awareness of external conditions and contingencies that could impact their 
systems and internal conditions and contingencies that could impact their neighbors' 
systems. Duri ng the 11-minute time span of the September 8th event entities observed 
changes i n flows i nto thei r systems, but were unable to understand the cause or 
significance of these changes and lacked sufficient time to take corrective actions. If 
affected entities had seen and run studies based on real-ti me external conditions prior to 
the event, they could have been better prepared to redispatch generation or take other 
control actions and deal with the i mparts when the event started. 

1 1 D, for example, is adjacent to APS, and the changes in flows on APS's system, 
especially on its 500 kV lines, can affect the flows on 1 1 D's system and vice versa. Yet, 
1 1 D's vi si bi I i ty i nto APS's system i s I i mi ted to i nformati on about the ti e I i ne between 
them. I n fart, 1 1 D's visi bi I ity i nto al I of its neighbors is I i mited to one or two buses 
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outside its system. 76 As a result I ID did not learn in real-ti me that H-NG tripped. IID 
also did not understand prior to the event how changes in flows or the loss of H-NG 
would affect its system. I mmediately after H-NG tripped, 1 1 D observed loading on its CV 
transformers escalate rapidly, but it had not been prepared for this escalation. 

I f 1 1 D had greater visi bi I ity i nto APS's system and 1 1 D had an equivalent on its 
RTCA that modeled the external network usi ng APS's real-ti me data i nstead of pseudo- 
generators modeled at the end of each tie line, 1 1 D's RTCA could have more accurately 
studied the results of a post-contingency loss of H-NG on its system before it occurred. 
After seeing the more accurate RTCA results, 1 1 D could have initiated appropriate 
control actions before H-NG tripped. Also, having real -time status of the H-NG would 
have better prepared 1 1 D to deal with the effects of its loss i n real ti me. 

I n addition to 1 1 D not having adequate situational awareness of APS's system, the 
affected TOPs and BAs external to 1 1 D were not aware i n real ti me of the effect of the 
post-conti ngency I oss of 1 1 D's three 230/ 92 kV transformers on thei r systems. Losses of 
the CV and Ramon transformers can cause SOL violations on neighboring systems. 
I ndeed, on September 8th, these transformer outages had a significant rippleeffect and 
led to the cascading nature of the event. Yet, entities outside 1 1 D's footprint were not 
prepared for these outages and, except for WECC RC, were unaware of the outages i n 
real ti me because of a lack of adequate visi bi I ity i nto 1 1 D's system. For example, at the 
ti me of the event, CAI SO's visibility into 1 1 D's system stopped at the ti e I i ne i nto 1 1 D's E I 
Centro station. 

The September 8th event exposed the negative consequences of TOPs havi ng 
I i mi ted external vi si bi I ity i nto nei ghbori ng systems. Provi di ng TOPs with the abi I ity to 
observe and model external system conditions and events on a continuous real-time 
basis will allow them to study and plan for the impact of external conditions and 
conti ngenci es before i t i s too I ate to react, as was the case on September 8th . 



^ IID has made efforts, even before the September 8th event, to receive more data points from adjacent utilities 
and is currently continuing this effort with all of its neighbors. 
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Finding 12 Inadequate Real-Time Tools: 

• Affected TOPs' real-time tools are not adequate or, in one case, operational 
to provide the situational awareness necessary to identify contingencies and 
reliably operate their systems. 

Recommendation 12: 

• TOPs should take measures to ensure that their real-time tools are adequate, 
operational, and run frequently enough to provide their operators the 
situational awareness necessary to identify and plan for contingencies and 
reliably operate their systems. 

Although many of the affected TOPs have and use real-ti me tools such as State 
Estimator and RTCA, some of the tools are not adequate or operational to provide the 
situational awareness necessary to effectively monitor and operate their systems. Also, 
some TOPs run or view these tools infrequently, whileothers run RTCA, for example, 
every five minutes. 

The alarming function on 1 1 D's RTCA provides an exampleof a real -time tool that 
does not adequately maximize situational awareness capabilities. 1 1 D's RTCA does not 
provide operators with any audible alarms or pop-up visual alerts when an overload is 
predicted to occur. I nstead, 1 1 D's RTCA uses color codes on a display that the operator 
must call up manually to learn of significant potential contingencies. For example, 1 1 D's 
RTCA might show that on the next contingency, a specific element will become 
overloaded. H owever, as currently designed, the operator must go to the specific page 
related to this element to view this result. The result will be color coded on this page, but 
this code does not function as an alarm. 

This design feature of 1 1 D's RTCA had negative consequences on the day of the 
event. Forty-four minutes prior to the loss of H-NG, 1 1 D's RTCA results showed that the 
N-l contingency loss of the first CV transformer would result in overloading of the 
second CV transformer to its tripping point. I f 1 1 D had taken action at this pre- 
contingency stage, it could have avoided the loss of both transformers. The 1 1 D operator, 
however, did not view the appropriate RTCA display and, therefore, was not alerted to 
the need to take acti on . If the operator had revi ewed the RTCA results and taken 
necessary corrective actions, he could have relieved loading on the transformers at this 
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pre-event stage, and thus mitigated the severe effects on the CV transformers that 
resulted when H-NG tripped. 77 

One affected entity, APS, has State Estimator and RTCA capability, but neither 
tool is operational. As a result, APS has limited capability to monitor and operate its 
system to withstand potential real-time contingencies. I nstead of using RTCA, APS 
relies on a set of previously studied contingencies and pre-determined plans to mitigate 
them. These studies are included in a manual that is created annually and usually 
updated several times a year. 78 By relying on pre-determined studies, APS cannot 
account and prepare for all potential contingency scenarios in real time. RTCA would 
provide APS with a more realistic analysis of its next potential contingency because the 
RTCA analysis is based on real-time conditions, as measured by State Estimator. 
Without RTCA, APS operators are not ful ly prepared to identify and plan for the next 
most critical contingency on its system. 

RTCA would have al lowed APS operators to study the i mpact of the loss of its H - 
NG. Although APS could have studied this contingency in its manual and seasonal 
studies, it could not have studied it based on real-time operating conditions that only 
State Estimator can provide. For example, APS's manual and seasonal studies did not 
study the loss of H-NG together with the multiple generator outages that existed on the 
day of the event. 79 As a result, APS was unprepared for the actual consequences of 
losing H-NG on September 8, 2011 including overloads on 1 1 D's 92 kV system and 
potential difficulty reclosing H-NG due to large phase angle differences. 80 



Since the event, II D has initiated changes to its RTCA program. First, it is working with a vendor to install an 
audible alarm feature. Second, IID has instructed its operators to constantly leave the RTCA result display screen 
on, rather than periodically calling it up. 

APS can also ask WECC RC or an APS engineer for a current-day study, but it usually relies on its manual for 
operations. APS also relies on WECC RC to notify it of any major post-contingency issues detected by WECC RC's 
RTCA results, but WECC RC might not consistently and promptly notify individual TOPs of all major issues. 

APS has indicated that it has had difficulty obtaining generator outage information from other BAs due to 
market and/or tariff concerns. 

Prior to the event, APS had been working with a vendor to build its RTCA capability and, since the event; it has 
accelerated its efforts to make RTCA operational. 
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Finding 13 Reliance on Post-Contingency Mitigation Plans: 

• One affected TOP operated in an unsecured N-lstate on September 8, 2011, 
when it relied on post-contingency mitigation plans for its internal 
contingencies and subsequent overload and tripping, while assuming there 
would be sufficient time to mitigate the contingencies. Post-contingency 
mitigation plans are not viable under all circumstances, such as when 
equipment trips on overload relay protection that prevents operators from 
taking timely control actions. If thisTOP had used pre-contingency 
measures on September 8th, such as dispatching additional generation, to 
mitigate first contingency emergency overloads for its internal 

conti ngencies, the cascadi ng outages that were triggered by the loss of H -NG 
might have been avoided with the prevailing system conditions on 
September 8, 2011 

Recommendation 13: 

• TOPs should review existing operating processes and procedures to ensure 
that post-contingency mitigation plans reflect the time necessary to take 
mitigating actions, including control actions, to return the system to a secure 
N-lstate as soon as possible but no longer than 30 minutes following a single 
conti ngency. As part of this review, TOPs should consider the effect of relays 
that automatically isolate facilities without providing operators sufficient 
time to take mitigating measures. 

Before September 8, 2011 1 1 D consistently relied on post-contingency mitigation 
plans, rather than proactively responding on a pre-contingency basis, for RTCA results 
showing that the N-l loss of one CV transformer would result in overloading on the 
second CV transformer. Post-conti ngency plans can work to prevent a second 
contingency as long as operators have sufficient time to take mitigating actions. Post- 
contingency mitigation is not an appropriate choice for theCV transformers, which are 
set to tri p by overload protection relays without al lowi ng operators enough ti me to take 
mitigating actions. Specifically, the transformers' overload protection scheme is set with 
a thin margin between the emergency rating and the relay trip point. The normal rating 
of the transformers is 150 MVA, the emergency rating is 165 MVA, and the relay trip 
point is set at 190.5 MVA, or 127% of the normal rating. Thus, when the transformers 
reach thei r emergency rati ng, operators may have the mistaken bel ief that they have 
sufficient time to take mitigating actions, when, in fact, the operators will have very little 
ti me before the transformers wi 1 1 tri p off I i ne, because they wi 1 1 soon reach the rel ay tri p 
setting. As shown below, pre-contingency mitigation measures are necessary when 
operators are faced with settings that leave such little margin between the emergency 
rating and overload trip point. 

On multiple days during the summer of 2011 1 1 D's RTCA results showed that an 
N-l conti ngency tripping of one of the CV transformers would result in overloading on 
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the second transformer. II D continued to operate in this state on multiple days without 
taking any pre-contingency mitigating actions. For example, 1 1 D did not dispatch 
additional generation on a pre-contingency basis to control the loading on oneCV 
transformer to prevent overloadi ng on the second CV transformer. There were 
potentially severe consequences of not taking pre-contingency actions. Specifically, 1 1 D's 
next-day study for September 8th detai led that the loss of both CV transformers would 
overload: (1) II D's Ramon transformer to its trip point; and (2) the S Line, which, in 
turn, would cause theS Line RAS to trip generation in Mexico that supplies power to the 
I mperial Valley substation. I n short, on multiple days in summer 2011 1 1 D's RTCA 
results showed that the loss of one CV transformer would overload the second 
transformer, and 1 1 D's next-day study revealed the cascadi ng outages that would stem 
from the loss of both transformers. Yet, 1 1 D did not institute pre-contingency mitigating 
measures, such as dispatching additional generation. 

I nstead, 1 1 D relied on post-contingency plans. On most days in summer 20 IX the 
I evel of overl oadi ng on the CV transformers gave 1 1 D j ust enough ti me to successful ly use 
a post-contingency mitigation plan to start generation after the loss of thefirst 
transformer to avoid the loss of the second transformer. H owever, on at least two days 
observed by the inquiry, a post-contingency plan would not allow the operator enough 
time to implement necessary procedures to mitigate the problem. On those two days, the 
loading on both CV transformers was high enough that only pre-contingency mitigation 
measures could have prevented the loss of the second transformer upon the loss of the 
first. On the first of those two days, I ID was simply fortunate that the N-l contingency 
I oss of the f i rst transformer never occurred . The second of the two days was September 
8, 2011 

Forty-four minutes prior to the loss of H-NG, 1 1 D's RTCA results showed that the 
N-l contingency loss of thefirst CV transformer would result in overloading of the 
second transformer to approxi mately 139% of its normal rati ng— leadi ng to the loss of 
the transformer by relay action. If IID had taken action at this pre-contingency stage, 
1 1 D might have been able to avoid the loss of both transformers. 81 After H-NG tripped, 
the rel ays took I ess than 40 seconds to tri p both CV transformers. Operators had no 
time to mitigate the overloads before the transformers were removed from service. 



OA The inquiry understands that the IID operator did not see these RTCA results and, thus, would not have known 
of the need for pre-contingency mitigating measures. There is no indication, however, that IID would have used 
pre-contingency measures regardless of the results. IID consistently relied on post-contingency measures. 
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Finding 1A WECC RC Staffing Concerns: 

• WECC RC staffs a total of four operators at any one time to meet the 
functional requirements of an RC, including continuous monitoring, 
conducting studies, and giving directives. The September 8th event raises 
concerns that WECC RC's staffing is not adequate to respond to emergency 
conditions. 

Recommendation 14: 

• WECC RC should evaluate the effectiveness of its staffing level, training and 
tools. Based on the results of this evaluation, it should determine what 
actions are necessary to perform its functions appropriately as the RC and 
address any identified deficiencies. 

WECC RC performs its reliability coordination functions through two offices. 
Although each office is capable of monitoring the entire Interconnection, during normal 
operations the offices have pri mary responsi bi I ity for monitori ng different parts of the 
Western I nterconnection. WECC RC's Vancouver, Washington, office is primarily 
responsible for monitoring the Pacific Northwest (excluding PacifiCorp East), California, 
and CFE's territory in Mexico. WECC RC's Loveland, Colorado, office is primarily 
responsible for monitoring the Desert Southwest area, Rocky Mountain area, 
PacifiCorp's East area, Sierra Pacific Power Company's area, 1 1 D's area, and the Los 
Angeles i ntermountai n area. Each office staffs two on-shift operators at al I ti mes. Each 
center dedicates an operator to the real-ti me desk (real-ti me operator) and the other 
operator to the study desk (study desk operator). 

The real-ti me operator's pri mary responsi bi I ities i ncl ude monitori ng I i mits and 
operating parameters, identifying exceedances, evaluating mitigation plans, and 
di recti ng corrective acti ons. The study desk operator's pri mary responsi bi I iti es i ncl ude 
monitoring expected post-contingency conditions to identify potential exceedances, 
evaluating actions being taken, and directing corrective action as necessary. The study 
desk operator also reviews WECC RC's next-day study for accuracy, conducts real-time 
studies to evaluate system conditions, and monitors EMS applications, such as RTCA, to 
identify any performance issues and request corrective actions, as necessary. The real- 
ti me operator and study desk operator also havesomejoint responsibilities, including 
reporting events that impact the BPS, identifying events or system conditions that 
require notification to adjacent RCs, and monitoring and testing primary and backup 
internal communication systems. Through these responsibilities, WECC RC is 
responsible for the reliable operation of the BPS in the WECC footprint, and it has the 
ultimate authority to prevent or mitigate emergency operating situations in both next- 
day and real -time timeframes. 
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I n addition, WECC RC is responsible for providing information to the entities in 
its footprint includingthe53TOPsand 37BAs. Some of this information is provided 
over the telephone. During the event in addition to performi ng the many RC functions 
they are responsi ble for performi ng, the RC operators had to answer phone cal Is 
providing or seeking information on the disturbance. 

Given WECC RCs responsibility and authority, four total operators— two in each 
regional office— might not be sufficient to effectively perform its function, particularly 
during emergency conditions. Several examples from the September 8th event highlight 
this concern. 

First, after the loss of H-NG, many alarms began sounding in WECC RCs control 
rooms, as voltage dropped and facilities overloaded. With so many alarms sounding in 
an emergency situation, the real-time operator had a difficult time prioritizing which 
alarms to monitor. WECC RC has eight unique categories, or "buckets," of alarms within 
its EMS applications, grouped according to importance. Buckets land 2 contain the 
highest priority alarms. Bucket 1 includes all 500 and 345 kV circuit breaker status 
changes, frequency and Path violations, status of generators greater than 50 MW and 
associated circuit breakers, and critical bus voltages. Bucket 2 includes all 220/230 kV 
ci rcuit breaker status changes and automatic voltage regulator status. 82 Buckets 3 
through 8 include lesser priority items, such as RAS status changes, non-critical bus 
voltages, and circuit breaker status changes below 220 kV. Operators receive audible 
alarms for buckets land 2 and typically leave bucket Is display on the screen constantly 
and use one other screen to display al I other buckets. 1 1 is a constant process to 
continually monitor the alarms, even during normal operating conditions, and it might 
not be possible for one real-time operator to keep track of and prioritize multiple alarms 
sounding at once. Also, both operators had numerous phone cal Is to field from entities 
throughout the affected areas, reporting and requesting information. Overburdening the 
real -time operator in this way could undermine his or her ability to perform the critical 
functions of monitoring system conditions and directing necessary corrective actions. 
Accordingly, WECC RC should consider whether additional operators are necessary to 
adequately perform these functions. 

A second indication that the current RC staffing levels might not be sufficient 
came duri ng the September 8th event when the study desk operator had to abandon his 
duties in order to provide support to the real-time operator by fielding phone cal I sand 



The CV 230/92 kV transformers are included in bucket 2. 
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monitoring conditions. On this day, the RC operators wereableto call for an engineer to 
conduct some studies. Because the September 8th event occurred during the afternoon, 
an engi neer was avai lable. F i ndi ng an engi neer to substitute for the study desk operator 
may not always be so easy. Late at night and early in the morning, no engineers are on 
duty. That the study desk operator needed to leave his responsibilities to support the 
real-ti me operator may i ndicate that one real-ti me operator and one study desk operator 
per office might not be sufficient to fulfil I WECCs reliability coordination functions. 

Alternatively, additional training and enhanced tools may enable an entity to 
accomplish more with the same number of personnel. While the inquiry observed a 
sampling of WECC RCs tools to be adequate during its site visit, WECC RC is in the best 
position to identify the combination of additional staff, enhanced tools, or training that 
best addresses the concerns identified by this report. 

Finding 15 Failure to Notify WECC RC and Neighboring TOPs Upon Losing 
RTCA: 

• On September 8, 2011, at least one affected TOP lost the ability to conduct 
RTCA more than 30 minutes prior to and throughout the course of the event 
due to the failure of its State Estimator to con verge. Theentitydid notnotify 
WECC RC or any of its neighboring TOPs, preventing this entity from 
regaining situational awareness. 

Recommendation 15: 

• TOPs should ensure procedures and training are in place to notify WECC RC 
and neighboring TOPs and BAs promptly after losing RTCA capabilities. 

When entities temporari ly lose thei r RTCA capabi I ity due to technical issues, they 
become bl i nd to the next most severe conti ngency on thei r system, and they do not know 
what pre-conti ngency measures might be necessary. Thus, when they lose RTCA, they 
must take i mmediate action to try to regai n thei r situational awareness. For example, 
after losing RTCA an entity should contact WECC RC, so the RC can monitor the entity's 
system and inform it of any significant issues. In such instances, theRC should also 
notify neighboring entities of any major contingencies that could impact their systems. 

Between 13:59 and the start of the event on September 8, 2011 WALC lost its 
RTCA when its State Estimator stopped solving. 83 As a result, WALC lost its ability to 
identify and study post-contingency violations and to take pre-conti ngency mitigating 
measures, as necessary. When it lost its RTCA, WALC should have contacted WECC RC 
and asked it to monitor WALC's area. WECC RC could have then notified WALC 



By not solving, or converging, the State Estimator stopped providing estimated values for the system. 
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regarding any significant problems and could have also contacted WALC's neighbors if it 
learned of any SOLs in WALC that were impacting the neighbors' systems- 84 Prior to 
the event on September 8, 2011 WALC experienced several post-contingency SOL 
violations, but without its RTCA capability, remained unaware of them. WECC RCs 
RTCA results showed these violations. WALC, however, did not notify WECC RC when it 
lost RTCA and, thus, WECC RC was unaware that it should notify WALC of the 
violations. An entity should never be operati ng i n an unknown state, as WALC was when 
it lacked functional RTCA and State Estimator, and did not ask any other entity to assist 
it with situational awareness. 

Finding 16 Discrepancies Between RTCA and Planning Models: 

• WECC's model used by TOPs to conduct RTCA studies is not consistent with 
WECC's planning model and produces conflicting solutions. 

Recommendation 16: 

• WECC should ensure consistencies in model parameters between its 
planning model and its RTCA model and should review all model parameters 
on a consistent basis to make sure discrepancies do not occur. 

The useful ness of RTCA study results and other real-ti me studies depend on the 
models used in the studies. Inaccurate models jeopardize the accuracy of studies, as well 
as entities' ability to respond appropriately to potential contingencies identified by the 
studi es. The i nqui ry's si mul ati on of the September 8th event di scovered that a 
discrepancy exists between WECC RCs model used to conduct RTCA studies and the 
model used for WECC's planning studies. Specifically, the impedance of 1 1 D's CV 
transformers differed by a factor of two between the WECC models. WECC's planni ng 
model has an impedance of 0.1 per unit, while WECC RCs RTCA model has an 
impedance of 0.05 per unit. This difference resulted in an error of approximately 16% in 
the RTCA model compared to the planning model with respect to loading on theCV 
transformers. 

Although theinquiry did not perform a comprehensive comparison of all 
parameters in WECC's various models, this discrepancy between the RTCA and planning 
models on such i mportant faci I ities cal Is i nto question the validity of other parameters i n 
WECC's models. 



°* While not at issue in this event, the RC should also notify TOPs if it loses its RTCA, so that TOPs know that the 
RC is not able to observe their systems. 
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I. System Analysis 
Consideration of BES Equipment 



■ Background 

The BES is generally defined as all facilities operating at voltages above 100 kV, 
although certain sub- 100 kV facilities with a significant impact on the BPS may be 
considered a part of the BES. Each RE currently determines its specific procedure for 
determining what is or is not BES. If a facility is not considered BES, relevant TOPs, 
BAs, and RCs may not study and model the i mpact of that faci I ity. 

Finding 17 Impact of Sub- IPO kV Facilities on BPS Reliability: 

• WECC RC and affected TOPs and BAs do not consistently recognize the 
adverse impact sub- 100 kV facilities can have on BPS reliability. As a result, 
sub- 100 kV facilities might not be designated as part of the BES, which can 
leave entities unable to address the reliability impact they can have in the 
planning and operations time horizons. If, prior to September 8, 2011, 
certain sub- 100 kV facilities had been designated as part of the BES and, as a 
result, were incorporated into the TOPs' and RCs planning and operations 
studies, or otherwise had been incorporated into these studies, cascading 
outages may have been avoided on the day of the event. 

Recommendation 17: 

• WECC, as the RE, should lead other entities, including TOPs and BAs, to 
ensure that all facilities that can adversely impact BPS reliability are either 
designated as part of the BES or otherwise incorporated into planning and 
operations studies and actively monitored and alarmed in RTCA systems. 

WECC RC, as well as TOPs and BAs impacted by the event did not consider 1 1 D's 
92 kV network and facilities ( including the CV and Ramon transformers) as BES 
elements. 1 1 D did not reconsider whether the CV and Ramon transformers should be 
studied like BES facilities even after a draft study sponsored byCFE (and shared with 
1 1 D) suggested the existence of a through-flow issue between the 500 kV substations at 
Devers and I mperial Valley adversely impacting 1 1 D's 92 kV network (including the CV 
and Ramon transformers) during contingencies on BPS systems, including H-NG. 85 
Because the Rel iabi I ity Standards apply to BES faci I i ties, if the CV transformers had been 
considered BES facilities, IID would have been required to study the impact they could 



See CFE's Path 45 Increase Rating Phase 2 Study Report, January 12, 2011, at 19. 
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have on BPS reliability. 86 Also, WECC RC and the affected TOPs would likely have 
i ncl uded the f aci lities in their studi es and been aware of the i mpact the I oss of H - N G 
would have on 1 1 D's 92 kV system, as wel I as the i mpact various tri ps withi n 1 1 D's 92 kV 
system would have on the rest of the BPS. The i nqui ry determi ned that duri ng the 
event approximately 12% (168 MW) of the original flow on H-NG was transferred 
through 1 1 D's 92 kV system, maki ng the 92 kV system part of a bul k power path as wel I 
as a significant looped transmission facility. The cascading outages that resulted from 
the loss of H-NG demonstrated the significant potential for 1 1 D's 92 kV system, including 
the CV transformers, to i mpact BPS rel iabi I ity. 

IROL Derivations 
■ Background 

In order to ensure the reliable operation of the BPS, entities are required to 
identify and plan for I ROLs, which are SOLs that, if violated, can cause instability, 
uncontrolled separation, and cascading outages. Once an I ROL is identified, system 
operators are then required to create plans to mitigate the impact of exceeding such a 
limit to maintain system reliability. 

Finding 18 Failure to Establish Valid SOLs and Identify I ROLs: 

• The cascading nature of the event that led to uncontrolled separation of San 
Diego, I ID, Yuma, and CFE indicates that an IROL was violated on 
September 8, 2011, even though WECC RC did not recognize any I ROLs in 
existence on that day. I n addition, the established SOL of 2,200 MW on Path 
44 and 1,800 MW on H-NG are invalid for the present infrastructure, as 
demonstrated by the event. 

Recommendation 18. L 

• WECC RC should recognize that I ROLs do exist on its system and, thus, 
should study I ROLs in the day-ahead timeframe and monitor potential I ROL 
exceedances in real-time. 



00 See, e.g., NERC Reliability Standard T0P-002-2b R11 (TOPs "shall perform seasonal, next-day, and current-day 
Bulk Electric System studies to determine SOLs"). 
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Recommendation 18.2: 

• WECC RC should work with TOPsto consider whether any SOLs in the 
Western I interconnection constitute I ROLs. As part of this effort, WECC RC 
should: (1) work with affected TOPs to consider whether Path 44 and H -NG 
should be recognized as I ROLs; and (2) validate existing SOLs, and ensure 
that they take into account all transmission and generation facilities and 
protection systems that impact BPS reliability. 

TheNERC Glossary defines an IROL as an SOL that if violated, could expose a 
widespread area of the BPS to instability, uncontrolled separation, or cascading outages 
that adversely impact the reliability of the BPS. Each IROL is associated with a 
maximum time limit (Tv) that the IROL can be exceeded before the risk to the 
I nterconnection or another RC area becomes greater than acceptable. The time limit can 
vary but any I ROL'sTv must be less than or equal to 30 minutes. 87 

For this event, the loss of H-NG should have been associated with an I ROL with a 
Tv for this N-l contingency of essentially minutes, because the cascading from the loss 
of H-NG began within seconds. However, neither WECC RC nor any of the affected 
entities have previously identified this I ROL. The WECC region historically has 
maintained an operating philosophy of not recognizing I ROLs. 88 I nstead, entities in the 
WECC region believe that as long as they operate within the conditions they have 
studied, they will not face the risk of IROLsand will not need to calculate I ROLs. The 
September 8th event undermines this philosophy. 

Prior to the event, the WECC system was supplying loads in the various balancing 
authority areas i n the range of 85-95% of thei r recorded peak loads. The power flows on 
al I the Paths i n the WECC region were below thei r maxi mum rati ngs and voltages were 
within acceptable levels. I n particular, the two major transmission corridors into the 
blackout area, namely Path 44 and H-NG, were loaded respectively to X302 M W and 



8 As defined by the NERC Glossary of Terms, an IROL's Tv is "[t]he maximum time that an [IROL] can be violated 
before the risk to the interconnection or other Reliability Coordinator Area(s) becomes greater than acceptable. 
Each [IROL's] Tv shall be less than or equal to 30 minutes." NERC Glossary of Terms, February 8, 2012, at 26. 

00 

00 As described by WECC in a February 16, 2012 Webinar on its SOL Methodology revision, "The WECC operating 
philosophy is to operate only in conditions that have been studied. Therefore, under these normal operating 
conditions, there are never IROL conditions (only SOLs). An IROL condition may be created by the occurrence of 
one or more unanticipated contingencies. When this occurs, under WECC Reliability Standards, bulk electric 
system operators are required to resolve the IROL condition within 20 minutes (stability) or 30 minutes 
(thermal)." http://www.wecc.biz/awareness/Reliability/Documents /SQL Methodology Presentation 
02.16.2012.pdf (emphasis in original). 
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X372 MW. Compared to their maximum SOL ratings of 2,200 MW and X800 MW, 
these loadings represent 59% and 78% of their maximum ratings— well within current 
limits. Path 44 and H-NG ratings of 2,200 MW and X800 MW may be invalid for the 
present i nfrastructure because cascadi ng outages due to a si ngle conti ngency occurred at 
loadings well below the SOL ratings. 

Duri ng the 11-mi nute di sturbance, the si ngle conti ngency of the sudden loss of H - 
NG resulted i n a series of cascadi ng outages, with multi pie elements exceedi ng thei r 
applicable ratings and leading to a widespread blackout of the area. 

Accordingly, WECC RC should lead all relevant TOPs in the blackout area to 
study and report on the appropriateness of identifying Path 44 and H-NG as I ROL paths. 
WECC RC should similarly assess transfer Paths outside this blackout area to ensure that 
there are no other similar reliability issues in the Western I nterconnection. Existing 
operati ng processes and procedures should be reviewed to ensure corrective control 
capabi I ities are provided to system operators to enable them to return the system to a 
secure N-l state as soon as possible, but no longer than 30 minutes foil owing a si ngle 
contingency. 

WECC RC has a proposed new SOL M ethodology document (current effective 
date of J une4, 2012), which acknowledges the need to establish IROLs, and theRC's 
responsibility to monitor I ROLs. 89 It recognizes that "Stability SOLs may qualify as 
I ROLs depending on the potential consequences of exceeding the limit and the impact on 
BES reliability. WECC RC makes this determination by collaborating with TOPs to 
understand the nature of the stabi I ity SOL, understandi ng the conditi ons that result i n 
the establ i shment of the stabi I ity SOL, and deter mi ni ng the BES i mparts of exceedi ng 
the stabi I ity SOL." 90 WECC RC also has a proposed multi-step process for determining 
whether thermal or voltage SOLs are I ROLs. In general, WECC RC will look at whether 
potential I ROLs cause "Widespread Adverse System I mparts," or "potential cascading." 
"Widespread Adverse System I mparts" is defined as "loading of three or more additional 
BES Facilities beyond 125% of their applicable emergency thermal Facility Rating, or 
[t]hreeor more additional BES Facilities with bus voltages experiencing voltages less 
than 90%."91 "Potential cascading" is defined as "when studies indicate that a 



m See WECC System Operating Limits Methodology for the Operations Horizon, Version 6.1, available at 
http://www.wecc.biz/awareness/Reliability/WECC RC Operating Procedures/WECC FAC 011-EFFECTIVE DATE 6-4- 
2012 SOL Methodology for the Operations Horizon.pdf . 

90 Id. at 5. 
91 /d. at 6. 
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contingency results in severe loading on a Facility, triggering a chain reaction of Facility 
disconnection by relay action, equipment failure, or forced immediate manual 
disconnection of the Facility (for example, public safety concerns, or no time for the 
operator to implement mitigation actions)." 92 

Impact of Protection Systems on Event 



■ Protection System Coordination 

When an abnormal system condition is detected on the BPS, relay protection 
systems operate to isolate the problem while causing minimum disturbance to the power 
system. This requires the relay to be selective in determining which elements to 
interrupt. The only method of obtaining this selectivity is to perform coordination 
studies. The inquiry discovered that two TOs did not properly coordinate a protection 
system and a third TO implemented a protection scheme without performing any 
coordination studies at all. This lack of coordination of protection systems resulted in 
circuits unnecessarily being interrupted, which had an undesirable effect on BPS 
rel iabi I ity duri ng the September 8th event. 

Finding 19 Lack of Coordination of the S Line RAS: 

• Several TOs and TOPs did not properly coordinate a RAS by: (1) not 
performing coordination studies with the overload protection schemes on 
the facilities that the S Line RAS is designed to protect; and (2) not assessing 
the impact of setting relays to trip generation sources and a 230 kV 
transmission tie line prior to the operation of a single 163/92 kV 
transformer's overload protection. As a result, BES facilities were isolated in 
excess of those needed to maintain reliability, with adverse impact on BPS 
reliability. 

Recommendation 19: 

• The TOs and TOPs responsible for design and coordination of the S Line RAS 
should revisit its design basis and protection settings to ensure coordination 
with other protection systems in order to prevent adverse impact to the BPS, 
premature operation, and excessive isolation of facilities. TOs and TOPs 
should share any changes to the S Line RAS with TPs and PCs so that they 
can accurately reflect the S Line RAS when planning. 

Operation of theS Line RAS isolates facilities beyond what is necessary to ensure 
reliability. TheS Line RAS is a directional overload scheme, located at the Imperial 
Valley substation, which is jointly owned by SDG&E and II D. TheS Line RAS was 
origi nal ly i mplemented to protect the sole 230/ 161 kV transformer at El Centro from 



M Id. 
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overloads due to increased flow on theS Line. 93 At the time, this was the only transfer 
poi nt from the 230 kV I i ne to the 161 kV system, and subsequently the 92 kV system, i n 
1 1 D's southern area. After implementing this RAS, 1 1 D has since installed a 230/92 kV 
transformer at El Centra, providing another path from the 230 kV system to the lower 
voltage networks. 

1 1 D's current intention for the S Line RAS is to reduce loading on the S Line by 
tripping generation and, if insufficient to reduce flow, tripping the S Li neat I mperial 
Valley Substation before transformer overload protection operates to trip the 163/92 kV 
transformer at El Centre Tri pping the S Line before all owing the El Centra 163/92 kV 
transformer's overload protection totakeaction effectively results in the removal of the 
230 kV source at the El Centra substation, which normally feeds a 230/92 kV 
transformer and a 230/ 161 kV transformer. Thus, the design of the S Line RAS 
intentionally isolates networked BES facilities to mitigate an overload on a non-BES 
facility (El Centra 163/92 kV transformer) to support reliability of the local system. 
Whilethis action alone does not constitute mi scoordi nation, proper coordination of a 
RAS should take into account, through system studies, the potential impact on BPS 
reliability, including potential interaction with other RASs and protection systems. 

During the September 8th event, the S Line RAS operated as designed, in that it 
tripped when it reached the settings that 1 1 D had prescribed. However, if one considers 
the purpose of the S Line RAS, which was to protect theEl Centra transformer from 
overloads, the S Li ne RAS operated long before it was needed. At the ti me that the S 
Line RAS operated, the El Centra 163/92 transformer was only loaded to 38% of its 
normal rating, and its overload trip point is 178% of its normal rating. Thus, the El 
Centra 163/92 transformer could have carried at least four times as much load before the 
transformer's overload protection system would have operated. Even though the El 
Centra transformer that the S Line RAS was designed to protect was nowhere near 
overloading, the S Line RAS tripped important generation and a 230 kV line. This calls 
into question the coordination of theS Line RAS with the transformer overload 
protection systems at El Centre 

II D provided SDG&E with the S Line RAS settings to implement. 1 1 D did not 
perform any studies to coordinate the S Line RAS with 1 1 D's protection systems. SDG&E 
did some studies to verify that the RAS coordinated with SDG&E 's protection systems. 
There is no indication that the S Line RAS was coordinated with 1 1 D's transformer 
overload protection at the El Centra station at which the S Line terminates. At a 



The S Line RAS also serves as secondary protection for other II D facilities if a RAS on the Imperial Valley to 
Miguel 500 kV line fails to operate. 
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minimum, 1 1 D, SDG&E and CAISO (astheTOP for SDG&E) should work together to 
ensure the proper coordination of the S Line R AS. 

To make matters worse, during the September 8th event, San Diego was relying 
on generation at I mperial Valley from the south when the S Line RAS tripped that 
generation. Loss of the I mperial Val ley generation caused San Diego to pul I even more 
power from the north, increasing the loading on Path 44 and causing the SONGS 
separation scheme to further exceed its trip point. If not tripped by the S Line RAS, 
generation at I mperial Valley could have helped SDG&E survive after the operation of 
the SONGS separation scheme. The inquiry's simulation showed that, had the S Line 
RAS tripped only the S Line without tripping the generation, the SONGS separation 
scheme would not have operated, and only 1 1 D would have lost power. 94 

Finding 20 Lack of Coordination of the SONGS Separation Scheme: 

• SCE did not coordinate the SONGS separation scheme with other protection 
systems, including protection and turbine control systems on the two SONGS 
generators. As a result, SCE did not realize that Units 2 and 3 at SONGS 
would trip after operation of the separation scheme. 

Recommendation 20: 

• SCE should ensure that the SONGS separation scheme is coordinated with 
other protection schemes, such as the generation protection and turbine 
control systems on the units at SONGS and UFLS schemes. 

SCE, theTO and TOP of the SONGS separation scheme, did not perform any 
protection system coordination studies for the separation scheme it implemented at 
SONGS. The scheme is intended to isolate five 230 kV lines simultaneously if its preset 
value is exceeded for a sustained period. If SCE had coordinated the separation scheme 
with other protection and generation control systems at SONGS, it may have recognized 
the potential for the operation of the SONGS separation scheme to cause the SONGS 
generators to tri p. Coordi nati on i n thi s context requi res system studi es to assess the 
i mpact of operati on of the RAS on the power system, i ncl udi ng potenti al i nteracti on with 
other RASs and protection systems, such as UFLS schemes. 

I n addition to the consequences at SONGS itself, the lack of coordi nation of the 
systems means that, when the scheme operates, the system enters an unknown state. 
During the event, the operati on of the protection scheme significantly contributed to the 



See footnote 53. 
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blackout of SDG&E, CFE, and Yuma— an effect neither coordinated nor adequately 
studied prior to the event. The inquiry's simulation indicates that SDG&E, CFE and, 
Yuma would not have been blacked out if the SONGS separation scheme had not 
operated, with I i mited i mpact to the rest of the Western I nterconnection. 

Finding 21 Effect of SONGS Separation Scheme on SONGS Units: 

• The SONGS units tripped due to their turbine control systems detecting 
unacceptable acceleration following operation of the SONGS separation 
scheme. 

Recommendation 21: 

• GOs and GOPs should evaluate the sensitivity of the acceleration control 
functions in turbine control systems to verify that transient perturbations or 
fault conditions in the transmission system resulting in unit acceleration will 
not result in unit trip without allowing timefor protective devices to clear 
the fault on the transmission system. 

When the SONGS separation scheme operated, turbines at SONGS began to 
accelerate i n excess of thei r control system setti ng causi ng both units to tri p offl i ne. The 
tripping of the SONGS units in this manner raises questions about the sensitivity of the 
turbi ne control system's setti ngs. The units are expected to withstand severe faults on 
the transmission system and allow the transmission protection systems to operate 
without the generators tripping offline. The coordination required for this protection is 
not a traditional relay-to-relay coordination; rather, the setting for the acceleration 
function should be coordinated with capabilities of the turbine and with the system 
response anticipated following operation of transmission protection systems for faults 
under various system conditions. The setting should also be coordinated with the system 
response foil owing operation of the SONGS separation scheme. Had the turbine control 
system acceleration function been coordinated in this manner, the trip of the units may 
have been avoided. 
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Finding 22 Lack of Review and Studying I mpact of SPSs: 

• Although WECC equates SPSs with RASs, prior to October X 2011, WECC's 
definition of RAS excluded many protection systems that would be included 
within NERC's definition of SPS. Asa result, WECC did not review and 
assess all NERC-defined SPSs in its region, and WECC's TOPs did not 
perform the required review and assessment of all NERC-defined SPSs in 
their areas. 

Recommendation 22: 

• WECC RE, along with TOs, GOs, and Distribution Providers (DPs), should 
periodically review the purpose and impact of RASs, including Safety Nets 
and Local Area Protection Schemes, to ensure they are properly classified, 
are still necessary, serve their intended purposes, are coordinated properly 
with other protection systems, and do not have unintended consequences on 
reliability. WECC RE and the appropriate TOPs should promptly conduct 
these reviews for the SONGS separation scheme and the S Line RAS. 

TheNERC definition of an SPS concludes with "Also called Remedial Action 
Scheme." 95 This implies that all SPSs are RASs and vice versa, but prior to October X 
2011 the WECC region did not equate SPSs with RASs. 96 WECC created four 
classifications of protection systems that fal I under theNERC definition of SPS, and, 
instead of including all of these classifications in the RAS definition, WECC only 
identified a subset of those protection systems as RASs. Safety Nets, Wide Area 
Protection Systems (WAPS), and Local Area Protection Systems (LAPS) were excluded 
from the WECC definition of a RAS even though they are SPSs as defined by NERC. 

For example, SCE did not study the impact of the SONGS separation scheme on 
BPS reliability because it believed, by classifying this scheme as a Safety Net, that it was 
not required to be studied. SCE also did not submit the separation scheme to WECC for 
revi ew by the Remedi al Acti on Scheme Rel i abi I i ty Subcommittee ( RASRS) . The i nqui ry 
determined that the SONGS separation scheme is indeed an SPS/ RAS as defined by 
NERC, because it altered the BPS configuration by separating Path 44 and redistributing 
generation in the absence of any faulted equipment. WECC, SDG&E, and SCE did not 
study the impact that the SONGS separation scheme could have on BPS reliability and, 
thus, were unaware of its severe i mpact on the BPS when the scheme operated: blacki ng 
out SDG&E and CFE and leading to the loss of the SONGS generators. 



yD NERC Glossary of Terms, February 8, 2012, at 46. 

On October 1, 2011, WECC revised its definition of RAS to include Safety Nets and Local Area Protection 
Schemes. 
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Another protection system that did not get the necessary scruti ny due to WECC's 
narrow definition of RAS was the S Line RAS. The S Line is a 230 kV transmission line 
that serves as a major tie between SDG&E &IID. It runs from II D's and SDG&E'sjointly 
owned I mperial Valley station on one end to 1 1 D's El Centro station on the other. TheS 
Line RAS, as 1 1 D and SDG&E called it, was classified as a LAPS by WECC, which called it 
the "S Line Scheme." Thus, the RAS received no periodic assessments. Like the SONGS 
scheme, the S Li ne RAS appears to be a SPS/ RAS as defi ned by N E RC, because it i s an 
automatic protection system that took action other than isolati ng a faulted faci I ity by 
tripping generation in Mexico for loading on a tie line between SDG&E and I ID. 

The S L i ne RAS was i mpl emented for two reasons: ( 1) to protect 1 1 D's system 
from overload during an N-2 event at SDG&E 's M iguel substation; and (2) to protect 
1 1 D's lone 230/ 161kV transformer at El Centro from overloads due to generation 
additions at I mperial Valley substation. The inquiry questions whether the scheme is 
still necessary, as both of the concerns that originally triggered installation of theS Line 
RAS have been mitigated. 1 1 D added a new transformer bank at El Centro, mitigating 
the concern for overloads on the 230/ 161kV transformer. Also, reconfigurations at 
M iguel along with the modifications to a RAS at M iguel have mitigated the concern of 
adverse effects on 1 1 D's system as a result of an N-2 event at M iguel . Si nee LAPSs are 
not periodically reviewed, the arguably outdated S Line RAS was still active during the 
September 8th event, and its operation contributed to 1 1 D's uncontrolled separation and 
the operati on of the SON GS separati on scheme by tri ppi ng over 400 M W of generati on 
before the S Line itself tripped. At a minimum, SDG&E, 1 1 D and CAISO should 
participate in the review of the S Line RAS. 

The SPSs that operated during the event suggest that WECC's previous exclusion 
of certain NERC-defined SPSs from WECC's RAS definition had an adverse impact on 
BPS reliability. 

Finding 23 Effect of I nadvertent Operation of SONGS Separation Scheme 
on BPS Reliability: 

• The inquiry's simulation of the event shows that the inadvertent operation of 
the SONGS separation scheme under normal system operations could lead to 
a voltage collapse and blackout in the SDG&E areas under certain high load 
conditions. 
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Recommendation 23: 

• CAI SO and SCE should promptly verify that the inadvertent operation of the 
SONGS separation scheme does not pose an unacceptable risk to BPS 
reliability. Until this verification can be completed, they should consider all 
actions to minimize this risk, up to and including temporarily removing the 
SONGS separation scheme from service. 

Theinquiry conducted a simulation to evaluate what would happen if the SONGS 
separation scheme inadvertently operated during normal system operations (e.g., in the 
absence of any outages, overloads, or SOL violations). Based on this simulation, the 
inquiry determined that under certain high load conditions, the operation of the scheme 
could result in voltage col I apse and a blackout in SDG&E'sand CFE's territories. The 
inquiry conducted a voltage stability study using a Power- Voltage (P-V) curve to estimate 
the amount of SDG&E load that could reliably be supplied after an inadvertent operation 
of the SONGS separation scheme. The P-V curve below in Figure 16 demonstrates that 
such operation would lead to a voltage col I apse and a blackout in the SDG&E and CFE 
territories under certain high load conditions. 



Figure 16: PV Curve for North Gila 500 kV Bus 
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Specif i cally, the system i s most I i kely to col I apse when the SDG&E I oad exceeds 
3,500 M W. I n 2010, SDG&E's load exceeded this amount for 851hours, 97 meaning that 



SDG&E Annual Electric Balancing Authority Area and Planning, FERC Form No. 714(2010). 
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the system was exposed to a potential blackout for approximately 10% of the year. This 
shows the potential risk to BPS reliability during normal system operations as a result of 
the i nadvertent operati on of the SON GS separati on scheme. Accordi ngly, given the I ack 
of studies done on the scheme, the i nqui ry recommends that the i nadvertent operation of 
the SONGS separation scheme be reviewed promptly to ensure it does not pose an 
unacceptable risk to BPS reliability. Until this verification can be completed, CAISO and 
SCE should consider all actions needed to minimize this risk, up to and including 
temporari ly removi ng the scheme from servi ce. 

M oreover, if SCE and CAI SO were to deci de to temporari ly remove the scheme 
from the service, the inquiry does not believe that BPS reliability would be jeopardized. 
I ndeed, i nqui ry si mul ati ons conducted for the day of the event show that if the scheme 
had not operated, the system, with the exception of col lapses i n the 1 1 D and Yuma areas, 
would have stabilized with minor overloads in the area around SONGS, acceptable 
voltages in theSDG&E area, and sufficient reactive margins in the critical portion of 
SCE 's system. 

Finding 24 Not Recognizing Relay Settings When Establishing SOLs: 

• An affected TO did not properly establish the SOL for two transformers, as 
the SOL did not recognize that the most limiting elements (protective relays) 
were set to trip below the established emergency rating. Asa result, the 
transformers tripped prior to the facilities being loaded to their emergency 
ratings during the restoration process, which delayed the restoration of 
power to the Yuma load pocket. 

Recommendation 24: 

• TOs should reevaluate their facility ratings methodologies and 
implementation of the methodologies to ensure that their ratings are equal 
to the most limiting piece of equipment, including relay settings. No relay 
settings should be set below a facility's emergency rating. When the relay 
setting is determined to be the most limiting piece of equipment, 
consideration should be given to reviewing the setting to ensure that it does 
not unnecessarily restrict the transmission loadability. 

TOsarerequiredtodesignateandsharetheirfacilities'SOLs. An SOL isthe 
value that satisfies the most limiting element of a facility beyond which the system 
cannot operate reliably. The i nqui ry's relay loadabi I ity calculations show that APS fai led 
to properly establ ish the SOL for two of its 500/ 69 kV transformers i n North Gila, 
because the transformers' relay loadability or load limit was actually set below their 
emergency rati ngs. A faci I ity cannot operate above its relay load I i mit, as operation i n 
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excess of a load limit results in the facility being removed from service. Thus, these 
setti ngs prevented the TOP from taki ng advantage of the short term emergency rati ngs 
identified by the transformers' SOLs. These settings resulted in difficulties restoring 
power to the Yuma load pocket, as operators believed they could load the transformers 
up to thei r emergency rati ng. I nstead, the transformers tri pped below the emergency 
rating, delaying the restoration of power to Yuma. 

If the SOL derivation had considered the transformer relay load limit, theTO 
could have (1) provided an SOL that accurately reflected the relay load limit so the 
system operator could have limited the transformer loading appropriately, or (2) 
reviewed the relay load limit to determine whether it unnecessarily limited the 
transformer loadability, and if so, raised the transformer relay setting threshold above 
the transformer emergency rati ng whi I e coordi nati ng the setti ng with the transformer 
short-time thermal capability. 

Load-Responsive Phase Protection Systems Set Too Close to Normal or 
E mergency Rati ngs 

BES faci I ities at a mi ni mum are requi red to have normal and emergency rati ngs. 
The normal rati ng is a conti nuous rati ng or a rati ng that a faci I ity can be operated to on a 
daily basis that specifies the amount of electrical loading a faci I ity can support. The 
emergency rating specifies the level of electrical loading a faci I ity can support for a finite 
peri od of ti me. Operati ng a faci I ity beyond its normal and/ or emergency rati ng for an 
extended peri od of ti me wi 1 1 expose certai n equi pment i n that faci I ity to the ri sk of 
thermal damage. I n order to prevent thermal damage to facilities, someTOs implement 
overload protection systems that are designed to automatically isolate the faci I ities if 
operated beyond their emergency rating. 

A problem arises when overload protection systems are set in close proximity to a 
facility's normal or emergency ratings. Setting the overload protection close to the 
normal or emergency rati ngs restricts faci I ity loadi ng and prevents operators from 
having sufficient time to take remedial action to mitigate an overload before the faci I ity is 
automatically isolated by the overload protection system. 98 As the Commission stated 
in Order No. 733, "manual mitigation of thermal overloads is best left to system 
operators, who can take appropriate actions to support Reliable Operation of the Bulk- 



NERC Reliability Standard PRC-023-1 R1.11 provides the following guidance on setting of overload protection 
systems on transformers: "Set the relays to allow the transformer to be operated at an overload level of at least 
150% of the maximum applicable nameplate rating, or 115% of the highest operator established emergency 
transformer rating, whichever is greater." 
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Power System." 99 Protective relay settings limited transmission loadability with 
extremely conservative overload protection settings, resulting in cascading outages 
during the September 8th event. These settings resulted in facilities being automatically 
removed from service by relays before operators had an opportunity to take remedial 
action. 



Finding 25 Margin Between Overload Relay Protection Settings and 
E mergency Rati ng: 

• Some affected TOs set overload relay protection settings on transformers 
just above the transformers' emergency rating, resulting in facilities being 
automatically removed from service before TOPs have sufficient time to take 
control actions to mitigate the resulting overloads. OneTO in particular set 
its transformers' overload protection schemes with such narrow margins 
between the emergency rati ngs and the relay tri p setti ngs that the protective 
relays tripped the transformers following an N-lcontingency. 

Recommendation 25: 

• TOs should review their transformers' overload protection relay settings 
with their TOPs to ensure appropriate margins between relay settings and 
emergency ratings developed by TOPs. For example, TOs could consider 
usingthe settings of Reliability Standard PRC-023-1R. Ill even for those 
transformers not classified as BES. PRC-023-1R.L11 requires relays to be 
set to allow the transformer to be operated at an overload level of at least 
150% of the maximum applicable nameplate rating, or 115% of the highest 
operator established emergency transformer rating, whichever is greater. 

Relay loadability calculations indicate that the relay settings on a number of 
transmission faci I i ties I i mi ted transmission loadabi I ity to si ightly above the emergency 
rati ng. For example, the relays on 1 1 D's CV transformers were set to tri p at 127% of thei r 
normal rating. The parallel CV transformers were loaded to 130%, which was above their 
127% over I oad rel ay tri p poi nt, i mmed i atel y after the I oss of H - N G . Both transformers 
tripped less than 40 seconds later. If the transformers 7 overload trip point had been in 
accordance with PRC-023-1R.L1X the trip point would not have been exceeded 
immediately after the loss of the H-NG, and 1 1 D operators might have had time to take 
actions to prevent cascadi ng. 300 



yy Transmission Relay Loadability Reliability Standard, 130 FERC ^ 61,221, at P 212 (2010). 

^® II D originally used conservative settings because the CV transformers are rare, expensive, load-serving 
transformers. I ID has indicated, however, that it will increase the overload relay settings on the CV transformers 
to 150% of their normal rating, and will relocate an additional 230/92 kV transformer from another substation to 
CV. 
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During the September 8th event 1 1 D was unaware that the overload relay setting 
for the Ramon 230/ 92 kV transformer had been mistakenly set at 207% of its normal 
rati ng. 1 1 D i ntended the Ramon transformer to have been set to tri p at 120% of its 
normal rati ng. After the event, 1 1 D reduced the Ramon transformer's tri p setti ng from 
207% to 120%, making it more likely to trip during high-loading conditions or conditions 
similar to those that precipitated the blackout decreasing the opportunity for its 
operators to take mitigating actions during such conditions. This setting actually 
increased the risk of future cascading outages likethe one which occurred on September 
8, 2011 

Finding 26 Relay Settings and Proximity to Emergency Ratings: 

• Some TOs set relays to isolate facilities for loading conditions slightly above 
their thirty minute emergency ratings. As a result, several transmission 
lines and transformers tripped within seconds of exceeding their emergency 
ratings, leaving TOPs insufficient time to mitigate overloads. 

Recommendation 26: 

• TOs should evaluate load responsive relays on transmission lines and 
transformers to determine if the settings can be raised to provide more time 
for TOPs to take manual action to mitigate overloads that are within the 
short-time thermal capability of the equipment instead of allowing relays to 
prematurely isolate the transmission lines. If the settings cannot be raised 
to allow more time for TOPs to take manual action, TOPs must ensure that 
the settings are taken into account in developing facility ratings and that 
automatic isolation does not result in cascading outages. 

I n addition to the problematic protection settings of the CV transformers, which 
preci pi tated the cascade, the i nqui ry di scovered that several other faci I i ti es, i ncl udi ng a 
number of 1 1 D's 161 kV transmission I i nes and two of WALC's 163/ 69 kV transformers, 
had relay protection settings which were only slightly above those facilities' emergency 
rati ngs. These conservative setti ngs severely I i mited TOPs' response ti me before the 
faci I i ti es were i sol ated, preventi ng the operators from taki ng effective mi ti gati ng acti on 
agai nst the cascade. Whi le the i nqui ry did not determi ne whether less conservative relay 
settings on these other facilities would have mitigated the cascade, the applied settings 
nevertheless do not leave operators sufficient time to take mitigating steps to prevent or 
amel i orate the consequences of future events. 

Angular Separation 



When a transmission line trips or goes out of service, the phase angle will 
generally increase between its two terminal points. When angle differences become 
large, faci I ities connected to the system can lose synchronization, causi ng the system to 
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become unstable. Also, if the phase angle is too large, closing the line breaker back into 
service with a large angle difference may result i n damage to nearby generator turbi ne 
shafts, and the resulting power swings and oscillations could lead to system instability or 
collapse. To enable successful reclosing, studies should be run to determine the 
maximum phase angle difference allowable for a line to be closed back in and safeguards 
be put into pi ace to prevent reclosurewith excessive phase angle difference. Should the 
phase angle difference exceed the establ ished I i mit, generation or load must be adj usted 
to reduce it to the level that al lows the I i ne to be closed. 

Finding 27 Phase Angle Difference Following Loss of Transmission Line: 

• ATOP did not have tools in placeto determine the phase angle difference 
between the two terminals of its 500 kV line after the line tripped. Yet, it 
informed the RC and another TOP that the line would be restored quickly, 
when, in fact, this could not have been accomplished. 

Recommendation 27: 

• TOPs should have: (1) the tools necessary to determine phase angle 
differences following the loss of lines; and (2) mitigation and operating plans 
for reclosing lines with large phase angle differences. TOPs should also train 
operators to effectively respond to phase angle differences. These plans 
should be developed based on the seasonal and next-day contingency 
analyses that address the angular differences across opened system 
elements. 

The i nqui ry's si mulation shows that after H - NG tri pped, the voltage phase angle 
between the two termi nals i ncreased from 20 degrees to approxi mately 72 degrees. On 
the day of the event APS's synchro-check relay was set at 60 degrees, 101 meani ng APS 
would not have been able to recloseH-NG until it reduced the phase angle difference 
from 72 to 60 degrees, or changed the relay setti ng to al low the breaker to close. 
Specifically, the 60 degree setting would not have all owed APS to recloseH-NG until 
appropriate generation on both sides of North Gila was dispatched or load reductions in 
the areas west of N orth Gila were i mpl emented to reduce the difference of the voltage 
phase angle to 60 degrees. 



^ Based on additional studies, APS has since determined the maximum settings on its synchro-check relay at 
North Gila to allow a maximum phase angle difference of 75 degrees to reclose a line. To add margin, APS has 
implemented the relay setting at 70 degrees. 
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Although APS operators are trained to effectively respond to phase angle 
differences, 302 APS currently lacks the tools necessary to determine phase angle 
differences foil owing the I oss of a transmission line until the line is reenergized. 303 The 
training, therefore, does littlegood if the operators cannot determine whether a phase 
angle difference exists in thefirst place. Generally, APS operators can monitor phase 
angles through SCAD A, but in order to receive and review this data, the transmission 
line must be energized. After H-NG tripped, and prior to reenergizing the line, for 
example, APS had no way to know if the line could be reclosed within the permissive 60 
degree setti ng of its synchro-check relay. 1 1 lacked situational awareness of the phase 
angle difference. Yet, APS informed WECC RC and CAI SO that it believed the line could 
be reclosed quickly, when, in fact, thiscould not have been done due to the phase angle 
difference. 104 

To avoid a similar situation in the future, TOPs should ensure that they have 
adequate tools to determine phase angles after the loss of transmission lines. For 
example, they can install PM Us throughout their system, as APS plans to do, to increase 
their situational awareness of phase angles. Moreover, TOPs should ensure that their 
operators aretrained to respond to phase angle differences by, for example, 
redispatching generation. In addition, TOPs should not underestimate the time required 
to reclosea line, particularly without first knowing the phase angle difference. Here, for 
example, APS likely could not have reclosed the line quickly, even had it known the 
phase angle difference, given system conditions on the day of the event. 

I ndeed, the inquiry conducted a series of power flow simulations and found that 
significant amounts of generation redispatch were needed to close the phase angle 
difference. Figure 17, on the next page, shows the relationshi p between the voltage 
phase angle of H-NG as generation is redispatched between California and Arizona. The 
dispatched approach adjusts the avail able generation nearest the H assayampa and North 
Gila buses. As generation is dispatched to its maximum output in the vicinity of the two 



^ APS provides its certified operators with two training classes, Power System Dynamics and Dynamics of 
Disturbances, both of which address power angles and their ramifications. In addition, APS provides its new 
operator trainees with training on power angles. 

APS plans to expand its use of PMUs to enable it to determine phase angle differences even without a line 
being energized. Through the PMU data, APS would be able to determine voltage and angle measurements on 
live buses in its substations, through which it could calculate phase angle differences. 

304 APS did not intentionally mislead WECC RC and CAISO with this statement. Rather, it did not expect that 
there would have been such a large phase angle difference, as it had not previously experienced such a 
difference. Moreover, APS determined that the line was not damaged and, thus, it did not believe there would be 
any issues closing the line. 
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stati ons, other generators farther out are adj usted to effect the change i n voltage phase 
angles. 



Figure 17: Phase Angle of H-NG vs. Generation Shift 




The bl ue I i ne i n F i gure 17 i 1 1 ustrates that with the parti cul ar condi ti ons of the 
September 8th event approximately 1800 M W needed to be redispatched on both ends 
of H-NG (and close to the terminals, in Southern California and Arizona) in order to 
close the voltage phase angle from 72 degrees to 60 degrees (i.e., to within the permissive 
60 degree setting of the synchro-check relay). The green line shows that more 
generation— more than twice as much— must be redispatched if units are chosen in 
Northern California to close the angle between Hassayampa and North Gila. 

While system operators could redispatch generation from available spinning 
reserves or commit units in the Southern and/ or Northern California area, it is 
questionable how quickly X800 MW could be dispatched. 
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Appendix A: List of Acronyms Used in Report 



Arc 


Ml fcrd L,UI III Ul CIlUl 


APS 


Ariynn^i Pi ihl \r ^pr\/irp 
/-\i i ia r uui i V- jci vi V-C 


BA 


Ral^nri nn Ai ithnritv 

UUI Ul 1 V . 1 1 IU AAULI IUI 1 Ly 


DEC 
DC J 


D U 1 K CI a-Li 1 L DybLa 1 1 


BPS 


R 1 1 1 k^- Pnwpr ^\/Qtpm 

DUIIx rUVVCI +Dy jLCI 1 1 


CAISO 


f"* ^lifnrni^i 1 nrlpnpnrlpnt" ^\/Qtpm Onpr^fnr 1 nr 

v_Cll 1 1 Ul III d 1 1 IUC|Ju IUCI IL «jy jLCI 1 1 w|sJCI aLUI , 1 1 IV-. 


CFF 


C* nmiQinn Fprlpr^l Hp FlprtriHrtaH 

v_,UI 1 HblUI I rOJCI CII UC LlCXLI IvJUQvJ 


cv 


Cnarhpl 1 a Val 1 pv 


CMC 
C l v l »J 


F nprnx/ M ^n^npmpnt" Q\/c1"om 
t_ i ici yy i v i di idya i la il oyoLa 1 1 


r;n 


Cl. r\v~\ /-\v- ->b-r\ v~ 0\A/nQr 

vjfcrl la dLUI UWI la 


uur 


Ocl la dLUI vyUa dLUI 


H-NG 


APS'^ H R^RvainnR- North fnila SOD kV tran^nni^inn linp 

/V i j j i i ujjuyui i i kjla i v wi li i v_j 1 1 a i\ v l i ai i «ji i ii jji wi i i i i 


IEEE 
■ ^ ^ ^ 


1 n^titi ifp nf F Iprtrir^l and Flprtrnnir^ Fnninppr^ 

1 1 1 3LI LULC Ul 1— 1 V3V-LI 1 L.CII Cll IU 1— 1 CV- LI Ul 1 1 v_ J L_ 1 1 LJI 1 1 OT 3 


1 1 n 


Irnnpri^l Irri nation nictrirt" 
1 1 1 iua idi mi lydLiui i L>MbLi ila 


1 ROI 


I ntpr^onnpr+i on R pI i ;^hi 1 i Onpr^tinn 1 imit" 

I I iLa uui ii lOwLi ui i r\a i dui 1 1 Ly u|Jci dLi i iy i_ 1 1 1 n l 


kV 

IX. V 


Kilnvnlt 

IX 1 1 UVUI L 


LAPS 


1 oral Arpa Prntprtinn ^wtpm 

i_wL.cn ca r i ulclliui i >jy JLCI 1 1 


M\/A 


M pn^\/nlt"-^rnnprp 
i v i tydvui l cii i iua c 


MIAI 

|V| yy 


l v l fcrydWdLL 


MCDf 
IM C r\ V- 


IM Ul LI 1 Ml I ltrl 1 Ldl 1 C 1 cLLI 1 L r\al dUI 1 1 Ly l_.Ul pUl dLI Ul 1 




C* \ fornix/ IVI pyi cc\ Onpr^tionc Oh irl\/ Qi ihr^ornrnit+cv^ 

v_-dl 1 1 Ul 1 II d/ l v l CAI LU U|Ja dLI Ul lb OLUUy JUULUI 1 II 1 II LL03 


OTC 


Onprati nn Tran^pr Panrihi 1 iti p<^ 

uuci a li i iy i i cii 1 3i a v^auoui 1 1 li cj 


OTC PC 


Onpr nn Xr^nQfpr C* ^n^hilitv Pnlirv C* nmmit1~PP 
wpa ciLi i iy i i cii ioi ci v>d|jciui 1 1 Ly r ui i L.y v— ui i n i n llcc 


PC 


pi anni nn (""YinrHi npitnr 

r i cii ii hi iy V/Uui vj 1 1 icill/I 


PMU 


Ph PiQnr Mp^qi irpmpnt' 1 Init" 

r 1 ICOUI l v l COoU I a 1 ICI 1 L Ul ML 


R AQ 


Ixallculdl MLLIUll DLI la lie 


or 

l\v 


r\a I dUI 1 1 Ly v_uui Ul I idLUl 


DC 
l\ C 


rxcylUIIdl CIILILy 


RTCA 


Rpal-Ti mp Pnnti nnpnrv Anslv^i^ 

i \ cai i 1 1 1 ivi. v—wi i li i lya iv-y t\ i icii yoi 3 


SASG 


^ni ithwp^l - Arpa ^ti iHv fnrni in 

JUL! LI 1 VVvZ-DL r\ 1 CCl ^JLLJLJ y L_J 1 LVLJ kJ 


CfAHA 


^i inpn/iQnrw nntrnl ^nrl Pi^t^ Ami MQitinn 
oufja vioui y v_^ui ili ui cii iu l^cilci /~\L.yui3i liui i 


crc 


DUULI la 1 1 v-^dl 1 IUI 1 lid CUIbUI 1 


SDG&E 


San Diego Gas & Electric 


SE 


State Estimator 


SOE 


Sequence of Events 


SOL 


System Operating Limit 


SONGS 


San Onof re Nuclear Generating Station 


SPS 


Special Protection System 
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CRD 


jalLr\lvtrl rUVVtrl 


SWPL 


Soi jfhwp^t" Pnwpr 1 ink 

JWU LI 1 V V COL 1 UVVQ 1 1 1 1 IX 


TO 


1 1 Cll 131 1 IIOOlWl 1 WVVI Id 


TOP 


1 I al lbl I II bbl Ul 1 vjptrl aLUl 


TP 


Tr^ncmi qqi nn Pl^nn^r 
i i cii loi i iibbiui i r idi ii la 


UFLS 


1 1 nHprf rpni ipnrv 1 n^H ^hcvlHi nn 

\j i ivjd 1 1 lajljci i V- y i_ uou »ji i vzaj vj i i ilj 


WAR 


V Ul L-Ml l iptrl tr rxtjClLLI Vfcr 


WALC 


Western Area Power Administration - Lower Colorado 


WAPS 


Wide Area Protection System 


WECC 


Western E 1 ectri city Coordi nati ng Counci 1 


YCA 


Yuma Cogeneration Associates 
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Appendix B: Table of Findings and Recommendations 

The f ol I owi ng tabl e provi des a compl ete I i st of f i ndi ngs and correspondi ng 
recommendations, each of which are discussed in detail at Section IV of the 



report. 



NEXT-DAY PLANNING 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


Findinql- Failureto Conduct and 


Recommendation I All TOPs should conduct 


TOPs 


Share Next-Day Studies: Notallof 
the affected TOPs conduct next- 
day studies or share them with 
their neighbors and WECC RC As 
a result of fai 1 i ng to exchange 
studies, on September 8, 2011 
TOPs were not alerted to 
contingencies on neighboring 
systems that could i mpact thei r 
i nternal system and the need to 
plan for such contingencies. 


next- day studies and share the results with 
nei ghbori ng TOPs and the RC ( before the next 
day) to ensure that all contingencies that 
could i mpact the BPS are studi ed. 


Findinq2- Lack of Updated 


Recommendation 2: TOPs and BAs should 


TOPs, BAs, RCs 


Fytprnal Mph/v/nrkc: i n MPYt-Dax/ 

1 ALO 1 1 CI 1 1 M d_ V V \J\ ISO 1 1 1 1 M CAL Uay 

Q1~i iHw oHol c - \A/hon rnnrli ir~H nn 
jLUUy l v IUUclb. VVlltril LUl IUULU I iy 

next-day studi es, some affected 
TOPs use models for external 
networks that are not updated to 
reflect next-day operating 
conditions external to their 
systems, such as generation 
schedules and transmission 
outages. As a result, these TOPs' 
next-day studies do not adequately 
predi ct the i mpact of external 
conti ngencies on thei r systems or 
internal contingencies on external 
systems. 


pnc:i irpfhat thpi r npyt-rtav c±i iHipc: arp i inrtafpH 

CT 1 JU 1 \Z LI IQL LI ICI 1 1 ICAL UQy DLUU 1 CO Ol C U k^LJQLCV-J 

LU I til 1 cLL 1 \kzj\L -Uay Uptrl all 1 iy LUl IUI LI Ul lb 

external to thei r systems, such as generation 
and transmission outages and scheduled 
interchanges, which can significantly impact 
the operati on of thei r systems. TOPs and BAs 
should take the necessary steps, such as 
executing nondisclosure agreements, to allow 
the free exchange of next-day operati ons data 
between operating entities. Also, RCs should 
review the procedures in the region for 
coordinating next-day studies, ensure 
adequate data exchange among BAs and 
TOPs, and facilitate the next- day studies of 
BAs and TOPs. 


Findina 3 - Sub- 100 kV Facility 

1 1 1 1 \-A 1 1 1 *J — J KA KJ J-\J \J IX V 1 \A \_l 1 1 1.1 V J 


Rprommpndation 3" TOP^ and RC^ should 

1 \ Uwl 1 II 1 IV_I IUULI \Jl 1 —J ■ 1 1 «J Ul IU 1 \V^»J »JI IUUIU 


TOPs RCs 


Not Adequately Considered in 
Next- Day Studies: Inconductina 
next-day studies, some affected 
TOPs focus pri mari ly on theTOPs' 
internal SOLs and the need to stay 
within established Rated Path 
limits, without adequate 
consideration of some lower 
voltage facilities. As a result, these 
TOPs r i sk over 1 ooki ng f aci 1 i ti es 
that may become overloaded and 
i mpact the rel i abi 1 ity of the BPS. 
Si mi 1 arly, the RC does not study 
sub- 100 kV facilities that impact 
BPS reliability unless it has 


ensure that thei r next-day studi es i ncl ude al 1 
internal and external facilities (including 
those below 100 kV) that can impact BPS 
reliability. 
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specifically been alerted to issues 
with such faci 1 i ti es by i ndi vi dual 
TOPs or the RC has otherwise 
identified a particular sub- 100 kV 
faci 1 ity as affecti ng the BPS. 






Findinq4- Flawed Process for 


Recommendation 4: WECC RC should 


WECC RC 


Estimatinq Scheduled 
Interchanqes: WE CC RC's process 
for esti mati ng schedul ed 
interchanges is not adequate to 
ensure that such values are 
accurately reflected in its next-day 
studies. As a result, its next-day 
studies may not accurately predict 
actual power flows and 
conti ngency over 1 oads. 


i mprove its process for predicti ng 

i nterchanges i n the day-ahead ti meframe. 


SEASONAL PLANNING 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


Findinq5- Lack of Coordination 


Recommendation 5: WECC RE should ensure 


WECC RE, TOPS 


in Seasonal Planninq Process: 
The seasonal planning process in 
theWECC region lacks effective 
coordination. Specifically, the four 
WECC subregions do not 
adequately i ntegrate and 
coord i nate studies across the 
subregions, and no single entity is 
responsi bl e for ensuri ng a 
thorough seasonal planning 
process. 1 nstead of conducti ng a 
full contingency analysis based on 

si 1 of fha ci i hroni nnc' oh iH i oc fho 
all Ul LI lc bUUl cyiUl lb bLUUIcb, Lllc 

subregions rely on experience and 
engi neeri ng j udgment i n choosi ng 
which contingencies to discuss. As 
a result, individual TOPs may not 
identify contingencies in one 
subregion that may affect TOPs in 
the same or another subregion. 


better integration and coordination of the 
various subregions' seasonal studies for the 
entire WECC system. To ensure a thorough 
seasonal planning process, at a minimum, 
WECC RE should require a full contingency 
analysis of the enti re WECC system, usi ng one 
integrated seasonal study, and should identify 
and eliminate gaps between subregional 
studies. 1 ndividual TOPs should also conduct 
afull contingency analysisto identify 
conti ngenci es outsi de thei r own systems that 
can i mpact the rel i abi 1 ity of the BPS withi n 

Lllcll bybLcill dllU bl 1UUIU bl ldl c u lcl l bfcjdbUlldl 

studies with TOPs shown to affect or be 
affected by thei r conti ngenci es. 


Findinq 6 -External and Lower - 


Recommendation 6: TOPs should expand the 


TOPS 


Voltaqe Facilities Not Adequately 
Considered in Seasonal Planninq 
Process: Seasonal planninq 
studies do not adequately consider 
all facilities that may affect BPS 
reliability, including external 
facilities and lower-voltage 
facilities. 


focus of thei r seasonal pi anni ng to i ncl ude 
external facilities and internal and external 
sub- 100 kV facilities that impact BPS 
reliability. 


Findinq 7- Failureto Study 


Recommendation 7: TOPs should expand the 


TOPs 


Multiple Load Levels: TOPsdo 
not always run their individual 
seasonal planning studies based on 
the multiple WECC base cases 


cases on which they run their individual 
planning studies to include multi pie base 
cases, as well as generation maintenance 
outages and dispatch scenarios during high 
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(heavy and light load summer, 
heavy and light load winter, and 
hpaw ^nri nci) hut in^tfiad mav 

i iDQvy ok/i 1 1 ly / , kj\A l, ii ijlouu , i i lay 

focus on only one load level. Asa 
result, contingencies that occur 
during the shoulder seasons (or 
other load levels not studied) 
might be missed. 


load shoulder periods. 




Findinq8- Not Sharinq Overload 


Recommendation 8: TOPs should include in 


TOPs 


Relay Trip Settinqs: In the 
seasonal planning process, at least 
oneTOP did not share with 
neighboring TOPs overload relay 
tri p setti ngs on transformers and 
transmission lines that impacted 
external BPS systems. 


the i nformati on they share duri ng the seasonal 
planning process the overload relay trip 
settings on transformers and transmission 
1 i nes that i mpact the BPS, and separately 
i dentify those that have overl oad tri p setti ngs 
below 150% of their normal rating, or below 
115% of the highest emergency rati ng, 
whi chever of these two val ues i s greater. 


NEAR- AND LONG-TERM PLANNING 


FINDING 


RECOMMENDATION 


APPLICABLE 
HiN 111 ItLZi 


Findinq9- Gaps in Near- and 


Recommendation 9: WECC RE should take 


WECC RE, TOPS, 
TPs, PCs 


Lonq-Term Planninq Process: 
Gaps exist in WECC RE's, TPs' and 
PCs' processes for conducti ng 
near- and long-term planning 
studies, resulting in a lack of 
consideration for: (1) critical 
system conditions; (2) the impact 
of elements operated at less than 
100 kV on BPS reliability; and (3) 
the i nteraction of protection 
systems, as a consequence, tne 
affected entities did not identify 
duri ng the planni ng process that 
the 1 oss of a si ngl e 500 kV 
transmission line could potentially 
cause cascadi ng outages. Planni ng 
studies conducted between 2006 
and 2011 should have identified 
the critical conditions that existed 
on September 8th and proposed 
appropriate mitigation strategies. 


actions to mitigate these and any other 
identified gaps in the procedures for 
conducting near- and long-term planning 
studies. The September 8th event and other 
major events should be used to identify 
shortcomi ngs when devel opi ng val i d cases 
over the planning horizon and to identify flaws 
in the existing planning structure. WECC RE 
should then propose changes to improve the 
performance of planni ng studies on a 
suoregiondi- ana i nterconnecuon-wiae oasis 
and ensure a coordinated review of TPs' and 
PCs' studies. TOPs, TPs and PCs should 
develop study cases that cover critical system 
conditions over the planning horizon; consider 
the benefits and potenti al adverse effects of al 1 
protection systems, including RASs, Safety 
Nets (such as the SONGS separation scheme), 
and overload protection schemes; study the 
interaction of RASs and Safety Nets; and 
consider the impact of elements operated at 
less than 100 kVon BPS reliability. 


FindinqlO- Benchmarkinq 


Recommendation 10: WECC dynamic models 


TPs 


WECC Dynamic Models: The 
i nqui ry obtai ned a very good 
correlation between the 
simulations and the actual event 
until the SONGS separation 
scheme acti vated . After acti vati on 
of the scheme, however, neither 
the tri ppi ng of the SON GS units 
nor the system col 1 apse of SDG&E 
and CFE could be detected using 
WECC dynamic models because 


should be benchmarked by TPs agai nst actual 
data from the September 8th event to i mprove 
their conformity to actual system 
performance. In particular, i mprovements to 
model performance from validation would be 
hel pful i n analysis of under and/ or over 
frequency events i n the Western 
1 nterconnection and the stability of islanding 
scenarios in the SDG&E and CFE areas. 
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some of the el ements of the event 
are not explicitly included in those 
models. Sample simulations of the 
islanded region showed that by 
addi ng known detai Is from the 
actual event, including UFLS 
programs and automatic capacitor 
switching, the simulation and 
event become more closely al igned 
following activation of the SONGS 
separation scheme. 






SITUATIONAL AWARENESS 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


Findinqll- Lack of Real-Time 


Recommendation 11 TOPs should enqaqe in 


TOPs 


External Visibility: AffectedTOPs 
have limited real -time visibility 
outsi de thei r systems, typi cal ly 
monitoring only one external bus. 
As a result, they lack adequate 
situational awareness of external 
contingencies that could impact 
thei r systems. They also may not 
fully understand how internal 
contingencies could affect SOLs in 
their neighbors' systems. 


more real-ti me data shari ng to i ncrease thei r 
visibility and situational awareness of external 
conti ngenci es that cou 1 d i mpact the rel i abi 1 i ty 
of thei r systems. They shoul d obtai n suffi ci ent 
data to monitor significant external facilities 
in real time, especially those that are known to 
have a di rect beari ng on the rel i abi 1 ity of thei r 
system, and properly assess the i mpact of 
i nternal conti ngenci es on the SOLs of other 
TOPs. In addition, TOPs should review their 
real-ti me monitori ng tools, such as State 
Estimator and RTCA, to ensurethat such tools 
represent critical facilities needed for the 
reliable operation of the BPS. 


Findinql2- 1 nadequate Real- 


Recommendation 12: TOPs should take 


TOPs 


Time Tools: AffectedTOPs' real - 
time tools are not adequate or, in 
one case, operational to provide 
the situational awareness 
necessary to identify contingencies 
and rel i ably operate thei r systems. 


measures to ensurethat thei r real-ti me tools 
are adequate, operational, and run frequently 
enough to provide their operators the 
situational awareness necessary to identify 
and plan for contingencies and reliably 
operate thei r systems. 


FindinqB- Relianceon Post- 


Recommendation 13: TOPs should review 


TOPs 


Continqency Mitiqation Plans: 
One affected TOP operated in an 
unsecured N-lstateon September 
8, 2011 when it relied on post- 
contingency mitigation plans for 
its internal contingencies and 
subsequent overload and tripping, 
while assuming there would be 
suffi ci ent ti me to mi ti gate the 
contingencies. Post-contingency 
mitigation plans are not viable 
under all circumstances, such as 
when equi pment tri ps on overload 
relay protection that prevents 
operators from taki ng ti mely 
control actions. IfthisTOPhad 
used pre-contingency measures on 


existi ng operati ng processes and procedures to 
ensurethat post-contingency mitigation plans 
ref 1 ect the ti me necessary to take mi ti gati ng 
actions, including control actions, to return 
the system to a secure N - 1 state as soon as 
possible but no longer than 30 minutes 
f ol 1 owi ng a si ngl e conti ngency. As part of th i s 
review, TOPs should consider the effect of 
rel ays that automati cal 1 y i sol ate f aci 1 i ti es 
without provi di ng operators suffi ci ent ti me to 
take mitigating measures. 
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September 8th, such as 
dispatching additional generation, 
to mi ti gate f i rst conti ngency 
emergency overloads for its 
internal contingencies, the 
cascadi ng outages that were 
tri nnprpH hv thp 1 nf H - N fn 

LI 1 UUQ \Z\A VJy LI 1 \3 IUDJ \J\ 1 1 IM VJ 

might have been avoided with the 
prevai 1 i ng system condi ti ons on 
September 8, 2011 






Findinal4- WECC RCStaffina 


Recommendation 14: WECC RC should 


WECC RC 


Concerns: WECC RC staffs a total 
of four operators at any one ti me 
to meet the functional 
requirements of an RC, including 
continuous monitoring, 
rondurti no ^tudipc; andnivino 
directives. The September 8th 
event raises concerns that WECC 
RC's staff i ng is not adequate to 
respond to emergency conditions. 


eval uate the effecti veness of i ts staff i ng 1 evel , 
trai ni ng and tools. Based on the results of this 
evaluation, it should determine what actions 
are necessary to perform its functions 
aDorooriatelv as the RC and address anv 
idpntifipd dpfiripnrip^ 

1 \*i t^l 1 LI 1 1 v^LJ LJ \Z\ 1 V—l \ZI 1 V—l t-J. 


Findinql5- Failureto Notify 


Recommendation 15: TOPs should ensure 


TOPs 


WECC RCand Neiqhborinq TOPs 
Upon Losinq RTCA: On 
September 8, 2011 at least one 
affected TOP 1 ost the abi 1 i ty to 
conduct RTCA more than 30 
minutes prior to and throughout 
the course of the event due to the 
fai 1 1 j rp nf Statp Fc;ti matnr tn 

1 Gil 1 Ul C \Jl 1 Lj «_JLQLV3 1 3 LI 1 1 ICILWI LU 

converge. The entity did not notify 
WECC RC or any of its neighboring 
TOPs, preventing this entity from 
regaining situational awareness. 


procedures and training are in pi ace to notify 
WECC RC and neighboring TOPs and BAs 
promptly after losing RTCA capabilities. 


Findinql6- Discrepancies 


Recommendation 16: WECC should ensure 


WECC 


Between RTCA and Planninq 
Models: WECC's model used bv 
TOPs to conduct RTCA studies is 
not consistent with WECC's 
planning model and produces 
conflicting solutions. 


consistencies in model parameters between its 
planning model and its RTCA model and 
should review all model parameters on a 
consistent basis to make sure discrepancies do 
not occur. 


CONSII 


)ERATION OF BES EQUIPMENT 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


Findinql7- Impact of Sub- 100 


Recommendation 17: WECCastheRE 


WECC RE, TOPs, 
BAs 


kV Facilities on BPS Reliability: 
WECC RC and affected TOPs and 
BAs do not consistently recognize 
the adverse i mpact sub- 100 kV 
facilities can have on BPS 
reliability. Asa result, sub- 100 kV 
facilities might not be designated 


should lead other entities, including TOPs and 
BAs, to ensure that al 1 faci 1 i ti es that can 
adversely impact BPS reliability are either 
designated as part of the BES or otherwise 
incorporated into planning and operations 
studies and actively monitored and alarmed in 
RTCA systems. 
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as part of the BES, which can leave 
entities unable to address the 
rel i abi 1 i ty i mpact they can have i n 
the pi anni ng and operati ons ti me 
horizons. 1 f, prior to September 8, 
20U certain sub- 100 kV facilities 
had been designated as part of the 
BES and, as a result, were 
incorporated into the TOPs' and 
RC's planning and operations 
studies, or otherwise had been 
incorporated into these studies, 
cascadi ng outages may have been 
avoi ded on the day of the event. 






IROL DERIVATIONS 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


FindinqlS- Failureto Establish 


Recommendation 18.1 WECC RC should 


WECC RC, TOPs 


Valid SOLs and Identify 1 ROLs: 
The cascadi ng nature of the event 
that led to uncontrolled separation 
of San Diego, 1 1 D, Yuma, and CFE 
indicates that an 1 ROL was 
violated on September 8, 20U 
even though WECC RC did not 
recognize any 1 ROLs in existence 
on that day. 1 n addition, the 
established SOL of 2,200 MW on 
Path 44 and 1,800 MWonH-NG 
are i nval i d for the present 
infrastructure, as demonstrated by 
the event. 


recognize that 1 ROLs do exist on its system 
and, thus, should study 1 ROLs in the day- 
ahead timeframe and monitor potential 1 ROL 
exceedances in real-time. 

Recommendation 18.2: WECC RC should 


work with TOPs to consider whether any SOLs 
in the Western 1 nterconnection constitute 
IROLs. As part of this effort, WECC RC 
should: (1) work with affected TOPs to 
consider whether Path 44 and H-NG should be 
recognized as 1 ROLs; and (2) validate existing 
SOLs, and ensure that they take into account 
all transmission and generation facilities and 
protection systems that impact BPS reliability. 


PROTECTION SYSTEMS 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


Findinql9- Lack of Coordination 


Recommendation 19: TheTOs and TOPs 


TOs, TOPs 


of the S Li ne RAS: Several TOs 


responsible for design and coordination of the 
S Line RAS should revisit its design basis and 
protection settings to ensure coordination 
with other protection systems in order to 
prevent adverse i mpact to the BPS, premature 
operation, and excessive isolation of facilities. 
TOs and TOPs shoul d share any changes to the 
S Li ne RAS with TPs and PCs so that they can 
accurately reflect the S Li ne RAS when 
planning. 


and TOPs did not properly 
coordinate a RAS by: (1) not 
performing coordination studies 
with the overload protection 
schemes on the faci 1 i ti es that the S 
Line RAS is designed to protect; 
and (2) not assessi ng the i mpact of 
setti ng rel ays to tr i p generati on 
sources and a 230 kV transmission 
ti e 1 i ne pr i or to the operati on of a 
single 163/92 kV transformer's 
overload protection. As a result, 
BES facilities were isolated in 
excess of those needed to mai ntai n 
rel i abi 1 i ty, wi th adverse i mpact on 
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BPS reliability. 






Findinq 20 - Lack of 


Recommendation 20: SCE should ensure that 


SCE 


Coordination of the SONGS 
Separation Scheme: SCE did not 
coordinate the SONGS separation 
scheme with other protection 
systems, including protection and 
turbi ne control systems on the two 
SONGS generators. As a result, 
SCF did not rpal i 7P that I J nit^ 7 

Jv> 1 kJ 1 1 IUL 1 kZLA 1 1 LI IUL \J 1 1 1 Lj £— 

and 3 at SONGS would trip after 
operation of the separation 
scheme. 


the SONGS separation scheme is coordinated 
with other protection schemes, such as the 
generation protection and turbine control 
systems on the units at SONGS and UFLS 
schemes. 


Findinq 21- Effect of SONGS 


Recommendation 21: GOs and GOPs should 


GOs, GOPs 


Separation Scheme on SONGS 
Units: The SONGS units tripped 
di jp to thpi r ti jrhi np rontrol 

LJLIC LU LI IQ 1 LUI Yj\ 1 1 \Z v_WI ILI \Jl 

systems detecting unacceptable 
accel erati on f ol 1 owi ng operati on of 
the SONGS separation scheme. 


eval uate the sensi ti vi ty of the accel erati on 
control functions in turbine control systems to 
x/prifvthat tranc;ipnt nprturhationc; nr fault 

V \Z\ 1 1 y LI IQL LI Gil 131 d 1 L L/d LU 1 k/QLI \Jl 13 \Jl 1 CI LI 1 L 

conditions in the transmission system 
resulting in unit acceleration will not result in 
uni t tri p without al 1 owi ng ti me for protective 
devices to clear the fault on the transmission 
system. 


Findinq 22- Lack of Review and 


Recommendation 22: WECC RE, alonq with 


WECC RE, TOs, 
GOs, DPs, TOPs 


Studyi nq 1 mpact of SPSs: 
Although WECC equates SPSs with 
RASs, prior to October X 20 U 
WECCs definition of RAS 
excluded many protection systems 
that would be included within 
NERC's definition of SPS. Asa 
rpcijlt \I\IFCC did not rp\/ipw and 

1 CDU 1 L, V V 1 V_^V_^ \A 1 \A 1 1 UL 1 CV 1 \Zv V CI 1 ILJ 

assess all NERC-defined SPSs in 
its region, and WECCs TOPs did 
not perform the requi red review 
and assessment of al 1 N E RC- 
defined SPSs in their areas. 


TOs, GOs, and Distribution Providers (DPs), 
shoul d peri od i cal 1 y revi ew the pu r pose and 
i mpact of RASs, i ncl udi ng Safety N ets and 
Local Area Protection Schemes, to ensure they 
are properly classified, are still necessary, 
serve their intended purposes, are coordinated 
properly with other protection systems, and do 
nnt ha\/p unintpndpd ron^pnupnrpc; on 

1 IUL 1 laVC. HI II 1 1 LCI 1 \J \Z\-A LUI 1 JCL>| LI \Z\ \ \-\ZZ3 \J\ 1 

reliability. WECC RE and the appropriate 
TOPs should promptly conduct these reviews 
for the SON GS separati on scheme and the S 
Line RAS. 


Findinq 23- Effect of 1 nadvertent 


Recommendation 23: CAI SO and SCE should 


CAI SO, SCE 


Operation of SONGS Separation 
Scheme on BPS Reliability: The 
inquiry's simulation of the event 
shows that the i nadvertent 
ooeration of the SONGS 
separation scheme under normal 
system operations could lead to a 
voltage col 1 apse and bl ackout i n 
theSDG&E areas under certain 
high load conditions. 


promptly verify that the i nadvertent operati on 
of the SON GS separati on scheme does not 
pose an unacceptable risk to BPS reliability. 
Until this verification can be completed, they 
should consider all actions to minimize this 
risk, up to and including, temporarily 
removi ng the SON GS separati on scheme from 
service. 


Findinq 24- Not Recoqnizinq 


Recommendation 24: TOs should reevaluate 


TOs 


Relav SettinqsWhen Establishinq 
SOLs: An affected TO did not 
properly establ ish the SOL for two 
transformers, as the SOL did not 


their facility ratings methodologies and 
i mpl ementati on of the methodol ogi es to 
ensure that their ratings are equal to the most 
1 i miti ng pi ece of equi pment, i ncl udi ng relay 
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recognize that the most 1 i miti ng 
elements (protective relays) were 
set to tr i p bel ow the establ i shed 
emergency rati ng. As a result, the 
tran^formpr^ tri nnpri nri or to thp 
faci 1 i ties bei ng loaded to thei r 
emergency rati ngs duri ng the 
restoration process, which delayed 
the restoration of power to the 
Yuma load pocket. 


setti ngs. N o rel ay setti ngs shoul d be set bel ow 
a faci 1 i ty's emergency rati ng. When the rel ay 
setti ng i s determi ned to be the most 1 i mi ti ng 
oieceof eauioment consideration should be 
oivpn to rpvipwi no thp ^ptti no to pn^u rp that it 

Ul VU 1 Lu 1 V— V 1 V_ v V II |U 1.1 1 V_ JV^LL 1 1 |U LVy V_-l 1 1 V— 1.1 IUL 1 I. 

does not unnecessari ly restrict the 
transmission loadability. 




Findinq 25- Marain Between 


Recommendation 25: TOs should review their 


TOs, TOPs 


Overload Relay Protection 
Settinqsand Emerqency Ratinq: 
Some affected TOs set overload 
relay protection settings on 
transformers j ust above the 
transformers' emergency rating, 
resulting in facilities being 
automatically removed from 
servi ce before TOPs have suffi ci ent 
time to take control actions to 
miti gate the resulting overloads. 
OneTO in particular set its 
transformers' overload protection 
^chpmpc; with ^uch narrow maroin^ 

1 V_l 1 IV > VVI LI 1 _JV^|V_I 1 1 IUI 1 WW 1 1 IUI Vjl 1 1 _} 

between the emergency rati ngs 
and the rel ay tri p setti ngs that the 
protective relays tri pped the 
transformers foil owing an N-l 
conti ngency. 


transformers' overload protection relay 
setti ngs with thei r TOPs to ensure appropri ate 
margi ns between relay setti ngs and emergency 
ratings developed by TOPs. For example, TOs 
could consider using the setti ngs of Reliability 
Standard PRC-023-1R. Ill even for those 
transformers not classified as BES. PRC-023- 
1 R . 1 11 requ i res rel ays to be set to al 1 ow the 
transformer to be operated at an overload level 
of at 1 east 150% of the maxi mum appl i cabl e 
nameplate rating, or 115% of the highest 
operator established emergency transformer 
rating, whichever is greater. 


Findinq 26 -Relay Settinqsand 


Recommendation 26: TOs should evaluate 


TOs, TOPs 


Proximity to Emerqency Ratinqs: 
Some TOs set relays to isolate 
facilities for loading conditions 
si i ghtly above thei r thi rty mi nute 
emergency rati ngs. As a result, 
several transmission lines and 
transformers tripped within 
seconds of exceedi ng thei r 
emergency ratings, leaving TOPs 
i nsuff i ci ent ti me to miti gate 
overloads. 


load responsive relays on transmission lines 
and transformers to determi ne if the setti ngs 
can be rai sed to provi de more ti me for TOPs to 
take manual action to mitigate overloads that 
are within the short- time thermal capability of 
the equi pment i nstead of al 1 owi ng rel ays to 
prematurely isolate the transmission lines. If 
the setti ngs cannot be raised to al low more 
ti me for TOPs to take manual acti on, TOPs 
must ensure that the setti ngs are taken i nto 
account in developing facility rati ngs and that 
automatic isolation does not result in 
cascading outages. 


ANGULAR SEPARATION 


FINDING 


RECOMMENDATION 


APPLICABLE 
ENTITIES 


Findinq 27- PhaseAnqle 


Recommendation 27: TOPs should have: (1) 


TOPs 


Difference Followinq Loss of 
Transmission Line: ATOP did not 


the tools necessary to determine phase angle 
differences foil owing the loss of lines; and (2) 
mitigation and operating plans for reclosing 
lines with large phase angle differences. TOPs 
should also train operators to effectively 
respond to phase angle differences. These 
plans should be developed based on the 
seasonal and next-day conti ngency analyses 


have tools in pi ace to determine 
the phase angle difference between 
the two terminals of its 500 kV line 
after the line tri pped. Yet, it 
informed the RC and another TOP 
that the line would be restored 
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quickly, when, in fact, this could 
not have been accomplished. 



that address the angular differences across 
opened system elements. 
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Appendix C: Comparison of August 2003 and September 2011 

Blackouts 

On August 14, 2003, an estimated 50 million people throughout the Midwest and 
Northeast United States and Ontario, Canada, experienced an electric power blackout. A 
day later, thejoint U.S.-Canada Power System Outage Task Force began investigating the 
causes of the blackout and consideri ng ways to prevent such outages i n the future. The 
task force detailed its findings and recommendations in an April 2004 report. 105 A 
comparison of thefindings and recommendations in this April 2004 report and the 
instant report on the September 8, 2011 blackout reveals commonalities between the 
two events. 

Although the August 2003 and September 2011 blackouts were triggered by 
different initiating events— tree touches in 2003 compared to a switching error in 2011— 
both blackouts had common underlying causes. First, affected entities in both events did 
not conduct adequate long-term and operations planning studies necessary to 
understand vulnerabilities on their systems. Second, affected entities in both events had 
inadequate situational awareness leading up to and during the disturbances. I n addition 
to these two underlying causes, both events were exacerbated by protection system relays 
that tri pped faci I ities without al lowi ng operators sufficient ti me to take mitigati ng 
measures. These si mi larities are highl ighted below, with excerpts from both reports to 
i 1 1 ustrate sped f i c compar i sons. 

Inadequate Long-Term and Operations Planning 

The 2003 Blackout Report states that "FirstEnergy was not [operating its system 
securely] because the company had not conducted the long-term and operational 
planning studies needed to understand [certain] vulnerabilities and their operational 
implications." 106 Similarly, this inquiry's report found that several entities' operational 
and long-term studies did not adequately ensure the reliable operation of their systems. 
Specifically, both reports described relevant planning studies that: (1) did not 
adequately identify and study critical external facilities; (2) did not adequately analyze 
potential contingency scenarios; and (3) were based on inaccurate models and invalid 
system operating limits (SOLs). 



J " iJ Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations 
(U.S.-Canada Power System Outage Task Force: April 2004) (2003 Blackout Report). 

306 2003 Blackout Report at 23. 
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Inadequate Long Term and Operations Planning 



Issue 

I nsuff i ci ent Anal ysi s i n 
Seasonal Studies 



2003 Blackout 

"[T] he studies FirstEnergy relied on . 
. . were not robust, thorough, or up- 
to-date. This left FE's planners and 
operators with a deficient 
understandi ng of thei r system's 
capabilities and risks under a range 
of system conditions." (P. 39). 



2011 Blackout 

'TOPs do not always run their 
individual seasonal planning studies 
based on the multi pie WECC base cases 
(heavy and light load summer, heavy 
and light load winter, and heavy 
spring), but, instead, may focus on only 
one load level." (Finding 7) 



"FE's 2003 Summer Study focused 
pri mari I y on si ngle-conti ngency (N- 
1) events, and did not consider 
significant multi pie contingency 

losses and security Overall, the 

summer study posited less stressful 
system conditions than actually 
occurred August 14, 2003 (when 
load was well below historic peak 
demand)." (P39). 



"Seasonal planning studies do not 
adequately consider all facilities that 
may affect BPS reliability, i ncl udi ng 
external facilities and lower- voltage 
faci I ities."( Finding 6) 

"I n the seasonal planning process, at 
least oneTOP did not share with 
neighboring TOPs overload relay trip 
setti ngs on transformers and 
transmission lines that impacted 
external BPS systems." (Finding 8) 



I nadequate 
Identification and 
Study of Critical 
External Facilities 



"On August 14 four or five capacitor 
banks within the Cleveland-Akron 
area had been removed from service 

for routine inspection These 

static reactive power sources are 

i mportant for voltage support 

The unavai labi I ity of the critical 
reactive resources was not known to 
those outside of FirstEnergy." (PP. 
26-27). 

"N E RC pol i cy requi res that cri ti cal 
facilities be identified and that 
neighboring control areas and 
rel iabi I ity coord i nators be made 
aware of the status of those faci I i ti es 
to identify the impact of those 
conditions on their own facilities. 
However, FE never identified these 
capacitor banks as critical and so did 
not pass on status information to 
others." (P. 27). 



"Not all of the affected TOPs conduct 
next-day studies or share them with 

their neighbors and WECC RC TOPs 

were not alerted to conti ngencies on 
neighbori ng systems that could i mpact 
thei r i nternal system and the need to 
plan for such contingencies." (Finding 
1) 

"I n conducting next- day studies, some 
affected TOPs focus pri mari I y on the 
TOPs' i nternal SOLs and the need to 
stay within established Rated Path 
limits, without adequate consideration 
of some I ower vol tage faci I i ti es. " 
(Finding 3) 

"[I n conducting next- day studies,] . . . 
the RC does not study sub- 100 kV 
facilities that impact BPS reliability 
unless it has specifically been alerted to 
issues with such facilities by individual 
TOPs..." (Finding 3) 
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I naccurate Dynamic 'The after-the-fact models developed 
Models to simulate August 14 conditions and 

events found that the dynami c 
model i ng assumpti ons for generator 
and load power factors in regional 
planning and operating models were 
frequently inaccurate." (P. 160). 



". . . neither the tripping of the SONGS 
units nor the system col lapse of SDG&E 
and CFE could be detected using WECC 
dynami c model s because some of the 
elements of the event are not expl icitly 
included in those models." (Finding 
10) 



To mitigate these concerns, the 2003 Blackout Report recommended that "NERC 
should work with the regional reliability councils to establish regional power system 
models that enable the sharing of consistent and validated data among entities in the 

107 

region/' and "[ c] I arify criteria for identification of operationally critical facilities, and 

108 

improve dissemination of updated information on unplanned outages." This inquiry's 
report I i kewise recommends that entities cooperate and coord i nate more effectively 
across all planning horizons, especially by increasing visibility in both external systems 
and lower voltage facilities that could impact BPS reliability. 



I nadequate Situational Awareness 

The 2003 Blackout Report stated, "A principal cause of the August 14 blackout 
was a lack of situational awareness, which was in turn the result of inadequate reliability 
tools and backup capabilities." 109 Similarly, the instant inquiry determined that 
i nadequate real-ti me situational awareness contri buted to the cascadi ng outages. I n 
both events, for example, the affected entities' real-ti me monitori ng tools were not 
adequate to alert operators to system conditions and contingencies. Also, some of the 
affected entities in both events did not use their real -time tools to monitor system 
conditions. As a result of these situational awareness issues, affected entities in both 
events were not aware that they were no I onger operati ng i n a secure N - 1 state and were 
not alerted to the need to take corrective actions. 



Inadequate Situational Awareness 



Issue 
System Vi si bility 



2003 Blackout 
"M I SO [the Rel i abi I i ty Coordi nator] 
had interpretive and operational 
tools and a large amount of system 
data, but had a limited view of FE's 
system." (P. 67). 



2011 Blackout 
"Affected TOPs have I i mi ted real- 
ti me vi si bi I i ty outsi de thei r 
systems, typi cal ly monitori ng 
only one external bus. Asa 
result, they lack adequate 
situational awareness of external 
contingencies that could impact 



107 



108 



109 



2003 Blackout Report at 160. 
2003 Blackout Report at 3. 
2003 Blackout Report at 159. 
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Inadequate Situational Awareness 



Issue 



2003 Blackout 



I inadequate Real-Time 
Monitoring Tools 



Operating in an 
Unsecu re State 



"FE's operational monitoring 
equi pment was not adequate to alert 
F E 's operators regard i ng i mportant 
deviations in operating conditions 
and the need for corrective action." 
(P. 19). 

"FE's control room operators lost 
the alarm fundi on that provided 
audible and visual indications when 
a significant piece of equipment 
changed from an acceptable to a 
problematic condition." (P. 51). 

M I SO's incomplete tool set and the 
f ai I u r e to su ppl y i ts state esti mator 
with correct system data on August 
14 contri buted to the lack of 
situational awareness." (P. 159). 

"F E 's operators were not aware that 
the system was operati ng outsi de 
first contingency limits . . . because 
they did not conduct a contingency 
analysis." (P. 64). 

"M I SO's rel i abi I i ty coord i nators 
were usi ng non- real -ti me data to 
support real-ti me "flowgate" 
monitoring. This prevented M I SO 
from detecting an N-l security 
vi ol ati on i n F E 's system and from 
assi sti ng F E i n necessary rel i ef 
actions." (P. 19). 

"Si nee F E 's operators were not 
aware and did not recognize events 
as they were occurri ng, they took no 
actions to return the system to a 
reliablestate."(P.65). 



2011 Blackout 
thei r systems. They also may not 
fully understand how internal 
contingencies could affect SOLs 
in their neighbors' systems." 
(Finding 11) 

"Affected TOPs' real -time tools 
are not adequate or, i n one case, 
operational to provide the 
situational awareness necessary 
to identify contingencies and 
reliably operate their systems." 
(Finding 12) 

". . . a TOP lost the ability to 
conduct [ Real Ti me Conti ngency 
Analysis] RTCA more than 30 
minutes prior to and throughout 
the course of the event . . land] 
did not notify WECC RCor any of 
its neighboringTOPs... "(Finding 
15) 



'The cascadi ng nature of the 
event that led to uncontrolled 
separation of San Diego, 1 1 D, 
Yuma, and CFE indicates that an 
[interconnection reliability 
operating limit] I ROL was 
violated ...In addition, the 
established SOLs of 2,200 MW 
on Path 44 and 1800 MWonH- 
NG areinvalid... "(Finding 18) 

"One affected TOP operated in an 

unsecured N-l state when it 

relied on post-contingency 
mitigation plans for its internal 
contingencies and subsequent 
overloads and trips, while 
assuming there would be 
suffi ci ent ti me to mi ti gate the 
contingencies." (Finding 13) 



To remedy these weaknesses in situational awareness, the 2003 Blackout Report 
recommended that entities [e]valuate and adopt better real-time tools for operators and 
reliability coordinators. "110 Similarly, this inquiry's report recommends that operators 



130 



2003 Blackout Report at 159. 
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develop and effectively uti I ize the real-ti me tools at thei r disposal and i nd ude al I 
facilities that can impact BPS reliability. 

Protection Systems 

During both events, protection system settings exacerbated and accelerated the 
cascading nature of the outages. As stated in the 2003 Blackout Report zone3 relay 
settings "did not cause the blackout [but] it is certain that they greatly expanded and 
accel erated the spread of the cascade. " m Si mi I ar I y, I oad responsi ve rel ay setti ngs 
accelerated the September 8th cascade and effectively el i mi nated the wi ndow i n which 
operators could have taken mitigating actions. 



Issue 

Overly Conservative 
Relay Protection 
Settings 



Cascading Relay 
Overload Trips 



Relay Protection 
Acti ng Too Qui ckly to 
Allow System 
Operators to Take 
Action 



> rotection Systems 

2003 Blackout 

"A few I i nes have zone 3 setti ngs 
designed with overload margins 
close to the long-term emergency 

limit of the line Thus, it is 

possi bl e for a zone 3 rel ay to operate 
on line load or overload in extreme 
conti ngency conditions even i n the 
absence of a fault." (P. 80) 



"[B]ecausethesezone2 and 3 relays 
tripped after each line overloaded, 
these relays were the common mode 
of fai I ure that accelerated the 
geographic spread of the cascade." 
(P. 80) 



"[T] he speed of the zone 2 and 3 
operations across Ohio and 
Michigan eliminated any possibility 
. . . that either operator action or 
automatic intervention could have 
limited or mitigated the growing 
cascade." (P. 80). 



2011 Blackout 

"Some affected TOs set overload 
relay protection settings on 
transformers j ust above the 
transformers' emergency rating, 
resulting in facilities being 
automatically removed from 
service before TOPs have 
suffi ci ent ti me to take control 
actions . . . following an N-l 
conti ngency." (F i ndi ng 25) 

"Some TOs set relays to isolate 
facilities for loading conditions 
si i ghtly above thei r thi rty mi nute 
emergency rati ngs. As a result, 
several transmission lines and 
transformers tripped within 
seconds of exceedi ng thei r 
emergency rati ngs, leavi ng TOPs 
insufficient time to mitigate 
overloads." (Finding 26) 

"Some affected TOs set overload 
relay protection settings on 
transformers j ust above the 
transformers' emergency rating, 
resulting in facilities being 
automatically removed from 
service before TOPs have 
suffi ci ent ti me to take control 
actions..." (Finding 25) 

". . . several transmission lines 



^■2003 Blackout Report at 82. Zone 3 relays "provide breaker failure and relay backup for remote distance 
faults on a transmission line." Id. at 80. 
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rotection Systems 




Issue 



2003 Blackout 



2011 Blackout 



and transformers tri pped withi n 
seconds of exceedi ng thei r 
emergency rati ngs, leavi ng TOPs 
insufficient time to mitigate 
overloads. (Finding 26) 



After seei ng the consequences of conservative zone 3 setti ngs, the 2003 Blackout 
Report recommended that "[i Industry is to review zone 3 relays on lines of 230 kV and 
h i gher . " m Th i s i nqui ry's report si mi I ar I y recommends that Transmi ssi on Owners 
review their facilities' overload relay protection setti ngs to ensure the appropriate 
margi n between relay setti ngs and emergency rati ngs. 



2003 Blackout Report at 158. 



- 130- 



FERC/NERC Staff Report on the September 8, 2011 Blackout 

Appendix D: Benchmarking the Model 

I. Introduction and Background 

The inquiry's Modeling and Simulation Team replicated system conditions on 
September 8, 20 IX and the events leadi ng up to the blackout. The model reflects the 
state of the electric system before and during the event with the real power output of 
generators dispatched to the val ues recorded i n SCADA data. With any major event on 
the BPS, it is important to accurately model the system before and during the event in 
order to: (1) verify the Sequence of Events; (2) support reconciliation of disparate 
measurement data; and (3) simulate and evaluate hypothetical scenarios, or "what-if" 
scenarios. 

In order to ensure the accuracy of these tasks, the Modeling and Simulation Team 
benchmarked the model to recorded SCADA and PM U measurements usi ng the 
fol lowi ng guidel i nes. Key faci I i ties and i nterfaces i n the affected area were general I y 
benchmarked to wi thi n 5% or 10 M VA accuracy to the measured data. Generator 
reactive outputs were also checked agai nst recorded val ues to ensure that the 
representation of reactive power margin was reasonably accurate. The team also 
monitored most other facilities in the affected area to ensure that the flows and voltage 
were reasonably close to measured data. Many of these other facilities also met the same 
guidel i nes used to benchmark the key faci I ities and i nterfaces. 

The iterative process between benchmarki ng and case alteration has traditional ly 
been time-consuming. The team pursued methods that would ultimately decrease the 
amount of time spent benchmarking so that results could quickly be used to identify 
probl em areas i n the case and make appropr i ate adj ustments. Because the team recei ved 
SCADA and PM U measurement data from many sources and entities, the data was: (1) 
organized into a consistent format, useful for automated benchmarking; and (2) cross- 
checked and verified for accuracy. I n organizing the data, the team also considered how 
each data point would map back to both power flow and dynamics results. The team 
ultimately achieved a single process to: (1) import power flow results; (2) import 
dynamics results; (3) compare the results to measured data from many sources at 
various quasi -steady state times during the event; (4) export tables showing the 
percentage accuracy; and (5) export graphs showi ng the accuracy of the results rel ati ve 
to measured data throughout the event. 
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II. Discussion 

The locations and measurements that the team selected for benchmarking were 
naturally predicated on the available measurements. While the team compared each 
available data point to the model results, it did not benchmark the model to all available 
data poi nts. I nstead the team focused its benchmarki ng effort on a "study area" that 
included SDG&E, 1 1 D, the APS Yuma load pocket, and portions of CFE and SCE. The 
team gave preference to measurements that were avai lable i n multi pie data sources with 
some reasonable agreement between the different sources, and particular preference to 
those I ocati ons where P M U measurements were avai I abl e, because these measurements 
could also be benchmarked against a full dynamics simulation. 

Following each set of simulations, the team reviewed the benchmarking data both 
graphically and tabularly, and tuned the modeling case and simulation parameters in an 
attempt to bring the case closer to measured reality. The team would then re-run the 
simulation, and repeat this process. 

Custom I interfaces 

Even though the team selected the best possible set of benchmarking data, and a 
substanti al amount of work went i nto cal i brati ng the study area of the model i ng case to 
those measurements, inconsistencies between some data points persisted. These 
i nconsistencies arose due to the multitude of subtle setti ngs and parameters for 
equipment, such as a changed tap on a single transformer affecting reactive power flow. 
For this reason, the team developed "custom interfaces" to benchmark an aggregation of 
poi nts. I f an aggregated, model ed sub- system was very cl ose to the actual measurements 
for that system, then the simulation could be trusted to accurately reflect the system. For 
example, if reactive power flow was mi sal located to a pair of adjacent transformers 
sourci ng a sub- system, the specific reactive flow on each transformer may not be of 
particular importance to the model. However, the reactive flow to the aggregate load 
bei ng served by those transformers may have a significant i mpact on a neighbori ng sub- 
system, and be crucial to effective benchmarking. 

The custom i nterfaces were also defi ned so as to i ndicate the amount of flow i nto 
or across a particular sub-system. For example, the calculated flows at the "I I D North 92 
kV System" i nterface give an idea of the amount and nature of the load i n the northern 
1 1 D 92 kV system. The custom i nterfaces sel ected i ncl ude: 

• 1 1 D North 92 kV System: All transmission sources for the northern 1 1 D 92 kV system, including 
the 230/92 kV transformers at Coachel I a Valley and Ramon, the 16V 92 kV transformers at 
Coachella Valley and Avenue 58, and the 92 kV lines between the northern and southern 1 1 D 
systems. 
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1 1 D South 92 kV System: All transmission sources for the southern II D 92 kV system, including 
the 230 kV transformer at El Centro, the 163/92 kV transformers at El Centro and Niland, and the 
92 kV lines between the southern and northern 1 1 D systems. 

Yuma Pocket: I nterfaces between the Yuma area 69 kV system (including portions of both APS 
and WALC service territories) and higher-voltage systems, including the 500/ 69 kV transformers at 
N . Gi la, the 16 V 69 kV transformers at Gi I a, and the 161 kV I i ne from Pi lot Knob to Yucca. 
Southwest California Desert I m ports: All transmission sources into the 
1 1 D/SDG&E/CFE/Yuma area other than Path 44. 



WECC Path 44 



I I I I I 




SDG&E 




IID North 
92 kV System 

I 

IID South 
92 kV System 



"S" Line 



I I 
I I 

Yuma Pocket 



1 / 

cket / 
JLi-/ 



Southwest California 
Desert Imports 



WECC Path 45 



CFE 



Figure 1: Key Facilities and Interfaces 



Key Facilities and Interfaces 

The team chose key faci I i ti es and i nterfaces i n the affected area as a way to 
quickly evaluate the model before fine-tuning it on a more granular level. Thesekey 
facilities and interfaces were benchmarked to within 5% or 10 M VA accuracy to the 
measured data throughout the enti re event. The key faci I i ti es and i nterfaces are I i sted 
below. 



• WECC Path 44 

• Southwest Cal iforni a Desert I mports 

• IID Northern 92 kV System 

• Niland-Blythe 161 kV Transmission Line 

• IID Southern 92 kV System 

• Imperial Valley- El Centro 230 kV Transmission Line("S" Line) 

• Miguel -Imperial Valley 500 kV Transmission Line 

• Yuma Pocket 

• El Centro-Pilot Knob 161 kV Transmission Line 

• Pilot Knob- Knob 161 kV Transmission Line 

• Pilot Knob-Yucca 161 kV Transmission Line 

• J ulian Hinds-Mi rage 230 kV Transmission Line 

• J ulian Hinds- Eagle Mountain 230 kV Transmission Line 



- 133- 



FERC/NERC Staff Report on the September 8, 2011 Blackout 



III. Results 

The following graphs demonstrate the benchmarking results. Each plot gives 
both power flow (see 'TSS" in graph legend) 113 and dynamic simulation (see "DYD" in 
graph legend) 114 results at each selected time step, with the corresponding SCADA 
and/ or PM u measurement as available. I n some instances, known issues with 
measured data are annotated on the charts, such as SCADA measurement errors for 
Coachel I a Val ley duri ng the i nterval fol lowi ng the i nitiati ng event. 

The simulated MW values fol low the measurements more closely than the 
simulated MVAR values. This is due to complexity involved in tuning voltage at each bus 
due to incomplete data, such as unknown tap values on large transformers. Overall, the 
M VA val ues are withi n our bench marki ng gui del i nes. 

The team also provided a table that compares: (1) the base case at 15:27:00 to the 
measured data; and (2) the case just prior to the loss of the Coachel I a Valley 
transformers at 15:28: 11 to the measured data. This table does not compare the 
dynami cs val ues to the base case at 15: 27: 00 because the power f I ow base case was the 
foundation for the dynamics simulation, meaning the val ues would be equal. 



Time Sequence Simulation. 
Dynamics Data. 
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PATH 44 iMW 
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PATH 44 MVAR 
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El Centro- Pilot Knob 161 MW 

50 -i 




El Centro- Pilot Knob 161 MVAR 

200 -i 




15:25:55 15:27:22 15:2S:4S 15:30:14 15:31:41 15:33:07 15:34:34 15:36:00 15:37:26 15:3 3:53 15:40:19 
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Pilot Knob -Yucca 161 MW 

180 -| 




15:25:55 15:27:22 15:28:48 15:30:14 15:31:41 15:33:07 15:34:34 15:36:00 15:37:26 15:38:53 15:40:19 
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J Hinds -Mirage 230 MW 
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J Hinds - Eagle Mt 230 MVAR 

50 -i 



50 




-20 
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Key faci I i ti es and i interfaces i n the affected area were general I y bench marked to 
within 5% or 10 M VA accuracy to the measured data. 







Base Case 


15:28:11 


Key 

Facility/ 
Interface 


Type 


Measured 


Power 
Flow 
Simulation 


Delta 
(Value) 


Delta 

(%) 




Measured 


Power 
Flow 
Simulation 


Delta 
(Value) 


Delta (%) 


Dynamics 
Simulation 


WECC 
Path 44 


MVA 


1310.25 


1323.61 


-13.36 


-1.02% 




2453.57 


2471.50 


-17.93 


-0.73% 


2457.19 


MW 


1296.85 


1292.09 


4.77 


0.37% 




2434.48 


2452.40 


-17.92 


-0.74% 


2442.90 


MVAR 


-186.93 


-287.17 


100.24 


-53.63% 




-305.49 


-306.68 


1.19 


-0.39% 


-264.64 


Southwest 
California 
Desert 
Imports 


MVA 


1328.45 


1336.38 


-7.93 


-0.60% 




333.50 




-4.92% 347.90 


MW 


I OU I .00 


1307.75 


-6.40 


U.H-C7 /O 




Qin 7fi 

O IU./D 


316.68 


-5.92 


-1.90% 


310.03 


MVAR 

IVI V rAPi 


Pfifi Qfi 


275.16 


-8.19 


_Q f)7% 
O . U / /o 




-1 21 .02 


-148.82 


27.80 


-22.97% 


-157.84 


IID North 
92 kV 
System 


MVA 


475.15 


476.81 -1.66 


-0.35% 




416.84 


480.17 


63.32 115 


15.19% 116 


466.78 


MW 


471.60 


473.23 


-1.63 


-0.35% 




414.58 


477.61 


-63.04 


-15.21% 


464.39 


MVAR 


-58.01 


-58.31 


0.30 


-0.52% 




-43.43 


-49.46 


6.03 


-13.88% 


-47.15 


Niland - 
Blythe 
161 kV 
Line 


MVA 


67.49 


69.45 


-1.97 


-2.92% 




108.41 


118.34 


-9.93 


-9.16% 


117.37 


MW 


-65.13 


-65.99 


0.86 


-1.32% 




-96.10 


-100.93 


4.83 


-5.02% 


-101.43 


MVAR 


17.68 


21.67 


-3.99 


-22.56% 




50.17 


61.79 


-11.62 


-23.16% 


59.07 


IID South 
92 kV 
System 


MVA 


105.60 


110.97 


-5.37 


-5.08% 




114.63 


132.43 


17.80 117 


15.53% 118 


124.85 


MW 


104.67 


110.91 


-6.24 


-5.96% 




106.14 


114.84 


-8.70 


-8.20% 


108.87 


MVAR 


-14.03 


-3.73 


-10.30 


73.42% 




-43.29 


-65.93 


22.65 


-52.31% 


-61.12 


Imperial 
Valley - El 
Centro 
230 kV 
Line ("S" 
Line) 


MVA 


96.08 


105.20 








125.31 


119.52 






MW 


94.24 


104.76 


-10.53 


-11.17% 




-109.22 


-101.41 


-7.81 


7.15% 


302.90 


MVAR 


-18.75 


-9.58 


-9.17 


48.89% 




61.42 


63.26 


-1.83 


-2.98% 


0.05 


Miguel - 
Imperial 
Valley 500 
kV Line 


MVA 


1088.80 


1095.22 


-6.42 


-0.59% 




225.19 


214.12 


11.07 


4.91% 77.98 


MW 


-1087.30 


-1093.10 


5.80 


-0.53% 




-188.50 


-191.82 


3.32 


-1.76% 


77.31 


MVAR 


57.18 


68.13 


-10.94 


-19.14% 




-123.20 


-95.15 


-28.05 


22.77% 


-10.26 


Yuma 
Pocket 


MVA 


280.78 


281.61 


-0.83 


-0.30% 




279.55 


283.18 


-3.63 


-1.30% 282.79 


MW 


280.11 


281.41 


-1.30 


-0.46% 




278.63 


282.81 


-4.18 


-1.50% 


281.63 


MVAR 


-19.46 


-10.70 


-8.76 


45.00% 




-22.66 


-14.46 


-8.20 


36.20% 


-25.68 



Large differences due to SCADA measurement errors at Coachella Valley and Ramon 

136 id. 

U7 The team experienced difficulty in calibrating the MVAR flows in this area, but are generally confident in the 
benchmarking because the MW values are within 10 MW. The MVA differences in the model appear to increase 
during this event. The representation of the system in this area of the model appears to assume that the IID 
South 92 kV system is a load serving local network. However, the actual transmission system operates in parallel 
with the rest of the BPS. It was difficult to calibrate the flows at the 92 kV to 161 kV interfaces because of the 
differences between the representation of the system in the model versus the parallel nature of the actual 
system. 

138 id. 
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El Centro 
- Pilot 
Knob 161 
kV Line 


MVA 


9.15 


9.83 


-0.69 


-7.51 % 




18.27 


21.17 


-2.90 


-15.87% 18.51 


MW 


-8.90 


-8.60 


-0.30 


3.40% 




15.73 


17.04 


-1.31 


-8.33% 


16.10 


MVAR 


2.09 


4.76 


-2.68 


127.99% 




9.29 


12.56 


-3.27 


-35.17% 


9.14 


Pilot Knob 
- Knob 
161 kV 
Line 


MVA 


74.27 


68.50 


5.77 


7.76% 




135.67 


140.14 


-4.46 


-3.29% 141.54 


MW 


-74.06 


-66.62 


-7.44 


10.05% 




-120.25 


-116.14 


-4.11 


3.42% 


-118.13 


MVAR 


5.54 


15.94 


-10.40 


187.80% 




62.83 


78.42 


-15.59 


-24.81% 


77.96 


Pilot Knob 
- Yucca 
161 kV 
Line 


MVA 


47.34 


41.17 


6.18 


13.05% 




132.82 


128.54 4.28 


3.22% 


130.96 


MW 


46.92 


40.94 


5.98 


12.74% 




127.39 


121.67 


5.72 


4.49% 


122.63 


MVAR 


6.33 


4.30 


2.03 


32.04% 




-37.60 


-41 .46 


3.87 


-10.28% 


-45.95 


Julian 
Hinds - 
Mirage 
230 kv 
Line 


MVA 


291.92 


287.50 


4.42 


1.51% 




273.49 


276.69 


-3.21 


-1.17% 


276.32 


MW 


291.87 


287.49 


4.38 


1 .50% 




273.46 


276.67 


-3.21 


-1.18% 


276.29 


MVAR 


-5.45 


-2.12 


-3.33 


61.10% 




-3.98 


-3.48 


-0.49 


12.43% 


4.45 


Julian 
Hinds - 
Eagle 
Mountain 
230 kV 
Line 


MVA 


55.47 


56.59 


-1.12 


-2.01% 




71.70 


68.07 


3.63 


5.06% 66.96 


MW 


53.29 


55.35 


-2.07 


-3.88% 




71.06 


65.89 


5.18 


7.28% 


66.26 


MVAR 


-15.43 


-11.78 


-3.64 


23.62% 




9.56 


17.13 


-7.57 


-79.17% 


9.64 
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FERC Staff 

Office of E nforcement 

Heather Polzin 
J eremy M edovoy 
Catherine Col I ins 
Samuel Backfield 
Thomas Lemon 
CheriseOjo 

Office of Electric Reliability 

Alan Phung 
AlirezaGhassemian 
Boris Voynik 
David Burnham 
Eddy Lim 
Gilbert Lowe 
J acob Lucas 
J ohn Spivak 
Ken Githens 
Kent Davis 
Leonard Chamberlin 
Louise Nutter 
Mahmood Mirheydar 
Michel I eVeloso 
MonicaTaba 
Pablo Ovando 
Perry Served io 
SasanJ alali 
TerranceClingan 
Terrence Si mon 
Thomas Reina 
Victor Barry 

Office of Energy Policy & I n novation 

Mary Cain 



150 



FERC/NERC Staff Report on the September 8, 2011 Blackout 



NERC Staff 

Ben McMillan 
Bob Cummings 
Chris McM anus 
DaveNevius 

Dmitry Kosterev (Technical Consultant from BPA) 

Earl Shockley 

Ed Ruck 

EricAllen 

Greg Henry 

J ames Merlo 

J im Griffith 

J im Robinson 

J ulieTate 

Kimberly Mielcarek 

M ark Vastano 

PhilTatro 

Phil Winston (Technical Consultant from Southern Company) 
Roman Carter 
Terry Brinker 

Department of E nergy Liason 

J amesMcGlone 

Nuclear Regulatory Commission Staff 

Singh Matharu 
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